Category: VMware

Following on from the excitement of VMware Explore in Las Vegas and receiving my new VMUG Advantage licenses for VCF 9 I have been testing various deployment choices for a new VVF 9 environment in my lab using the GA release of vCenter (9.0.0.0.24755230).

When downloading the various files to populate the offline depot for my cloud installer appliance to consume I noticed that there were two versions of ESX, an older one in both .ISO form (installer) and .ZIP (offline bundle), and a newer one in only the offline bundle.

Here’s what the GA version files look like in the drop down when displaying the original 9.0.0.0 version:

The small drop down box on the top left of the VMware ESX panel can be used to select which version should be displayed, but as you will see below, when selecting the option for 9.0.0.0100 the only file available is an offline bundle.

I had decided to build a custom ISO including some of the VMware Flings (which I thought would also be required to get a working VCF 9 deployment in the future) and used the newer version of ESX (9.0.0.0100) to install my ESX servers prior to executing the VVF installer process.

Mostly this has been a trouble free assumption, other than the warning displayed by the VVF cloud installer that I was using an unexpected version of ESX on my target hosts. I didn’t have any real reason to doubt this process (because the newer version of ESX fixes some critical CVEs) and continued with setting up my environment successfully.

However, during several repeated installations of my Supervisor cluster I ran into issues where the spherelet VIB didn’t always uninstall/reinstall correctly, and was required to remediate the cluster against the ‘autogen-software-spec-1’ lifecycle manager image – which is autogenerated using the version of ESX and any vendor plugins or components used at installation time.

At this point I noticed that often the remediation might fail because it would skip each of the hosts due to a ‘supposed’ hardware incompatibility and not complete the rest of the remediation. These warnings can be silenced within the vCenter UI under cluster object, Monitor tab, vSAN, Skyline Health.

Silence those alerts if you’re not interested in maintaining compatibility with the HCL (because it’s a lab environment for instance).

In trying to chase down the further cause of the problem I saw the orange ribbon displayed on the Hardware Compatibility tab:

“Requested target version is not supported for the cluster.”

Select the cluster object, choose Updates tab and examine the Hardware Compatibility.

This seems strange, because the remediation will eventually complete anyway because I am not using the remediation option to force hardware compatibility before starting. The following screen shows the same message in a different position.

Why isn’t the offline-depot from the 9.0.0.0100 version actually supported, even though the auto-gen profile is created automatically and the offline-depot file is imported? It seems that this just comes down to the release dates of vCenter and the subsequent ESX update. You can see this in more detail by examining the following file on the vCenter appliance:

/var/log/vmware/vmware-updatemgr/vum-server/hcl_python_lib.log

Here are the relevant points which called out to me:

• Called to discover Hardware Compatibility List
• Unknown version 9.0.0-0100.24813472
• Cannot find target version (9.0.0-0100.24813472) in VCG

2025-09-17T04:30:13.461Z INFO report.hcl Called to discover HCL for hostId host-34 with target version 9.0.0-0100.24813472, vSanHclConstraints = True.
2025-09-17T04:30:13.462Z ERROR compatibility.releases Got exception while using cache.
Traceback (most recent call last):
  File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/releases.py", line 37, in getEsxiReleaseByVersion
    result = getCacheFactory().getReleasesCache().getByVersion(version)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/cache/release_cache.py", line 72, in getByVersion
    return ProductRelease(releaseId=1, version=version)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/vmware-updatemgr/python/hcl/compatibility/vvs/models/product.py", line 38, in __init__
    raise ValueError("Unknown version %s" % version)
ValueError: Unknown version 9.0.0-0100.24813472
2025-09-17T04:30:13.462Z ERROR report.hcl Cannot find target version (9.0.0-0100.24813472) in VCG.

This would on the face of it seem strange, because the software depot even includes the version which is already deployed to the cluster and already knows the version which it presumably extracted from the host(s) when they were added to the cluster for the first time.

Is it as simple as the vCenter version is older than the ESX release, and that Broadcom haven’t yet released a patch for vCenter which recognises new versions of ESX?

Seems like we just need to get vCenter to agree that it’s not using the latest data and to search online for the updated information.

Eventually I stumbled across the menu option to synchronise the hardware compatibility settings and the release data:

Main vCenter menu dropdown, Lifecycle Manager, Actions drop down, Compatibility Data, Sync

Returning again to /var/log/vmware/vmware-updatemgr/vum-server/hcl_python_lib.log

2025-09-17T10:32:39.412Z INFO __main__ Loading VVS database from file /storage/updatemgr/patch-store/vvs/vvs-consolidated-bundle-download.json
2025-09-17T10:32:39.966Z INFO __main__ VVS database json loaded in-memory
2025-09-17T10:32:39.966Z INFO compatibility.cache Creating CacheFactory...
2025-09-17T10:32:39.970Z INFO compatibility.cache.executor Datastore locked
2025-09-17T10:32:39.970Z INFO __main__ Loading releases into datastore
2025-09-17T10:32:39.970Z INFO __main__ Releases to be loaded: 9
2025-09-17T10:32:39.971Z INFO __main__ Releases loaded
2025-09-17T10:32:39.971Z INFO __main__ Loading cpuseries into datastore
2025-09-17T10:32:39.971Z INFO __main__ Cpuseries to be loaded: 97
2025-09-17T10:32:39.972Z INFO __main__ Cpuseries loaded
2025-09-17T10:32:39.972Z INFO __main__ Loading servers into datastore
2025-09-17T10:32:40.365Z INFO __main__ Servers to be loaded: 5636
2025-09-17T10:32:40.709Z INFO __main__ Servers loaded
2025-09-17T10:32:40.709Z INFO __main__ Loading devices into datastore
2025-09-17T10:32:41.539Z INFO __main__ Devices to be loaded: 7033
2025-09-17T10:32:42.361Z INFO __main__ Devices loaded
2025-09-17T10:32:42.361Z INFO __main__ Loading OEM vendors into datastore
2025-09-17T10:32:42.361Z INFO __main__ OEM vendors to be loaded: 7
2025-09-17T10:32:42.362Z INFO __main__ OEM vendors loaded
2025-09-17T10:32:42.362Z INFO __main__ Updating datastore time to 1758091862563
2025-09-17T10:32:42.513Z INFO compatibility.cache.executor Datastore unlocked

In this case perhaps the issue isn’t that the versions aren’t being published necessarily but that the compatibility cache doesn’t yet include build ‘9.0.0.0100.24813472’.

Never afraid to reference a William Lam article, I would question why this release is supported for VCF 9 as described here: https://williamlam.com/2025/07/applying-1st-esx-live-patch-using-vcf-9-0-operations.html but not actually within VVF yet?

A quick check through the /storage/updatemgr/patch-store/vvs/vvs-consolidated-bundle-download.json file on my vCenter doesn’t show any release matching that version – so perhaps the initial error was to assume that this patch was also eligible at all for VVF 9 (and not as it seems for VCF 9 only) at this stage.

Perhaps we’ll find out with the next minor or patch release of vCenter if this omission is resolved, but at present it would appear that the best way to avoid these remediation problems in VVF is to only use the GA 9.0.0.0 release of the ESX ISO file.

During my own deployment testing of VMware Cloud Foundation 9, and its smaller counterpart VMware vSphere Foundation 9, I came across the need to determine exactly how much space is required if you want to set up an offline depot to install your first fleet member. In offline/airgap or edge locations where connectivity to the internet is either too slow or completely inaccessible VMware provides a great solution to be able to prepare a portable depot which could either be a virtual machine or laptop operating close to where you are going to run the VCF installer.

Previously the Cloud Builder appliance was very large (~27GB) because the image required a lot of space just to store the various files which it will need for the installation. The new combined VMware Cloud Foundation Installer/SDDC Manager .OVA download in VCF 9 replaces this element with a much smaller 2.15GB file because it doesn’t by default include the other components which will be installed later, e.g. VCF Operations, VCF Automation etc.

Instead you can choose to create an offline depot on a laptop or virtual machine and use this to bootstrap the VMware Cloud Foundation Installer with those files instead of having it download everything from the Broadcom site using your download token. This scenario is far more likely in my own experience, as downloading tens of GB when reaching a greenfield site (for any solution) is variously difficult and can take hours.

The product documentation states that the installer appliance requires a minimum of 914GB storage if thick provisioned, but this post will cover what is the bare minimum in order to get a working VM which will deploy the whole VCF suite.

Doing it the William Lam way

William Lam has several excellent articles which describe the process of (1) enabling a simple HTTP server using Python which will serve the files that are imported into the SDDC Manager when it is first deployed, and (2) these will be useful in setting up a depot (perhaps you’ll use a VMUG Advantage entitlement to download the files if it’s for a lab environment)

How to deploy VCF 9 using VMUG Advantage licenses: https://williamlam.com/2025/07/how-to-deploy-vvf-vcf-9-0-using-vmug-advantage-vcp-vcf-certification-entitlement.html

Using HTTP protocol to host the depot files: https://williamlam.com/2025/06/using-http-with-vcf-9-0-installer-for-offline-depot.html

Creating a simple Python web server: https://williamlam.com/2025/01/quick-tip-easily-host-vmware-cloud-foundation-vcf-offline-depot-using-python-simplehttpserver-with-authentication.html

Disable 10Gbit ethernet adapter speed check: https://williamlam.com/2025/06/disable-10gbe-nic-pre-check-in-the-vcf-9-0-installer.html

How much space?!

However rather than duplicating any of this above guidance, this post concentrates on exactly how much space you’ll need for either a VCF 9 or VVF 9 based offline-depot, ideally stored on a laptop or virtual machine acting as a web server. It could be costly in disk space terms to host these files permanently on your laptop, but there’s no reason why you can’t use an external SSD for this purpose – the question is what size will I need?

Also worth considering is where you’re going to put the VCF Installer virtual machine when you’re getting ready to bootstrap a vSAN environment. In this case you’ll need to find a VMFS datastore which is large enough for the installer plus the data which will be imported into the VM itself. The rest of your disks are probably going to be cleared of any partitions so you can’t use those to store data.

A fully populated offline depot which is capable of serving both VVF and VCF products to the SDDC Manager will require ~56GB on disk when combined with the other parts of the depot. The thin provisioned VM will use approximately 80GB when the offline-depot files have been uploaded to it, so if you’ve got a VMFS partition on a 250GB SSD or NVMe disk that is local to your ESXi server this should be sufficient to hold the installer VM before you eventually migrate it to vSAN.

Here’s how the space breaks down into the two products, each is neatly defined within the user interface of the VMware Cloud Foundation Installer:

VMware vSphere Foundation 9

You will need a total of 16.67GB free space to store the three files (.ova and .iso) comprising the three elements stored in the offline-depot.

VMware Cloud Foundation 9

You will need a total of 52GB free space to store the nine files (.ova, .tar, .vlcm and .iso) comprising the seven application elements stored in the offline-depot.

So there you have it, the whole VVF/VCF stack can be installed from these binaries using the VMware Cloud Foundation Installer, and the only element that you’ll need to install onto your target ESXi server initially is the 2GB installer VM (which can optionally become the actual SDDC Manager for the cluster once complete).

This is a placeholder posting for ongoing updates as and when new updates/resolutions are found. It isn’t intended to provide any additional detail to the problems outlined but simply to document the areas where bugs or ‘gotchas’ are located.

I have recently acquired several Intel NUC 11th Generation (NUC11TNHv50L) for my lab/testing environment which are being deployed into an existing vSAN/NSX-T environment as a workload domain. The release of these latest NUCs seemed to have generated a lot of interest with different community members discussing the ideal fit with NSX-T (due to the dual 2.5 Gbit/s Intel I225-LM NICs which come in the Pro version), however there are a couple of limitations that make this not a smooth ride currently.

Community networking driver and workarounds

Out of the box these NUCs are not supported with VMware ESXi and rely upon the Community Networking Driver Fling. Therefore before purchasing these devices for your home lab be aware that this fling:

• Requires a custom ESXi image to be created which includes the Community Networking Driver
• Does not support jumbo frames (e.g. >1500 byte MTU) – which in my view prevents any serious use with the NSX-T Geneve protocol which is typically 1600 byte minimum
• Causes the network interface to become disconnected (link layer communication fails) if configured MTU is greater than 1500, which only recovers after a reboot
• Seems to cause a purple screen (PSOD) failure when the second NIC is connected (under undefined circumstances currently)

Currently I am overcoming the NSX-T frame size issue by using the Startech USB 3.1 1Gbit/s USB network adapters, but this requires an additional fling to be installed. As a compromise it’s not too bad, since there are two Thunderbolt/USB-C ports on these NUCs allow up to two additional 1Gbit/s interfaces to be attached. So I am configuring my ESXi hosts as:

1 x Onboard Intel I225-LM at 2.5 Gbit/s – dvSwitch 1 (Management, vSAN)

1 x StarTech USB 3.1 adapter at 1Gbit/s – dvSwitch 2 (NSX-T, vMotion)

Power off and shut down

In addition it seems that when ‘Shut down’ of an ESXi host is performed the system ignores the BIOS power setting (e.g. to remain off, or power on etc.) and will immediately restart the operating back to a running condition (almost as if a reboot instead of shut down were chosen). This is strange behaviour which needs further experimentation and makes shutting down your lab a lot more time consuming – however it can be worked around currently by:

1. Shut down the ESXi instances individually using host UI/vCenter
2. Watch the power light on the front panel (assuming no screen attached) – when the power light turns off for approximately 0.5s it is initiating the actual power off, prior to becoming turned back on again
3. At this point pull the power supply out of the back of the NUC and plug it back in a couple of seconds later – it will remain off instead of rebooting (even if the BIOS setting says on loss of power – power on)

It’s getting hot in here

Lack of fan speed and temperature within ESXi hardware sensors. This is not a new issue but despite the integrated 3D graphics which is now on-chip there still seems to be a lack of information exposed to the operating system (presumably by Intel). In my bookcase vSAN/NSX-T environment it’s becoming a ‘hot topic’ to say the least ;-). Both new and older NUCs are doing fine on the Balanced performance/fan speed setting, and do a good job of spinning up and down the fan whenever the CPU turbo feature engages (up to 4.1GHz on my units), but it would be good to be able to view this more empirically than just watching how many windows need to be opened!

Good resources to check out in all things NUC are William Lam and Florian Grehl.