Can you remediate an ESXi 9.1 host cluster with VCFMS management services shut down?

I asked myself this question whilst updating a test VVF 9.1 lab cluster recently, and couldn’t be sure whether the new lifecycle management and fleet management services are now instrumental to host remediation?

My simple attempt to prove this scenario is as follows:

  1. Create a new empty host cluster in vCenter and call it ‘esx-upgrade-target’ for instance
  2. Select the new cluster and click on the Updates tab. Edit the cluster, assign it the desired image build and vendor add-ons (if necessary)
  3. Save the image using a name like ‘autogen-software-spec-2’
  4. Now select the cluster which contains the hosts that need remediating, click on the Updates tab and Assign the new image to the cluster
  5. Stage the update to ALL of the hosts so that the source of the image cannot be dependent on any external location
  6. Shut down the VCFMS control-plane VM (it has 10GB RAM and 4 vCPU)
  7. When complete, use Power, Shutdown guest on the remaining three Kubernetes nodes in the VCFMS cluster – because the control-plane VM is powered down already they won’t restart automatically
  8. NB – If you’re having trouble with a shortage of lab resources, now is a good time in the lab to resize them from 16GB RAM to 12GB, and 12 vCPUs to 6 (only attempt this if you don’t want to maintain compliance with the default ‘large’ size, this will then match a ‘medium’ size all the time you maintain the configuration). ONLY do this in a lab environment, not production!
  9. Now remediate the host cluster in the usual way, evacuating workloads via vMotion, entering maintenance mode and rebooting the hosts sequentially
  10. Start up the VCFMS control-plane VM, and then after a couple of minutes start any worker VMs which are still powered down.

At this point we can be sure that even if the source of the downloaded ESXi images is actually located on the VCFMS lifecycle/fleet servers initially that the action of staging the update removes any dependency.

In conclusion, whilst it makes sense that the downloaded ESXi image files could be stored along with the other bundles in the new VCF/VVF 9.1 depot (new fleet and lifecycle services) but it’s not fully clear if this is the case in practice.

Broadcom’s documentation describing how you can deploy VVF 9.1 without VCFMS suggests that host lifecycle management may not yet be fully integrated into the new fleet and lifecycle management framework yet, but an alternative page contradicts this assumption because it says that host lifecycle is now integrated with the depot download process: https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/9-1/managing-host-and-cluster-lifecycle/working-with-vsphere-lifecycle-manager-depots.html

I intend to repeat the process without staging the images for remediation and keeping VCFMS shut down in order to test the point, however there seems to be insufficient information around whether the new management services have to remain online – my bet so far (if short on resources during host reboots) is to pre-stage the image and only begin remediation once this has been done.