How to clean up NSX Advanced Load Balancer following replacement of a failed Tanzu control plane node

How do I clean up a missing control plane node in the Avi load balancer console?

This post outlines an approach I used to solve a problem which has occurred in several environments I’ve worked in recently. I haven’t seen a similar set of instructions anywhere yet, but it doesn’t mean that they are the only way to solve the problem. Check with VMware Support if you’re having a production problem, don’t follow this guidance without properly understanding the type of problem which you’re experiencing.


If you have found this page because you’re stuck with a similar problem it is probably because one or more of your control plane nodes in a Tanzu Kubernetes Grid (TKG) cluster have failed and been replaced automatically leaving a broken IP pool entry in NSX Advanced Load Balancer user interface.

For example, you log in and find that one of your IP pools which define the control plan endpoints are offline (shown as 3/4 servers up).

Clicking into the cluster will provide further detail of the missing control plane endpoints

In this case, one of the existing control plane nodes (172.20.11.45) became frozen and went offline , eventually losing its DHCP lease before it could be converted into a permanent reservation. Tanzu’s vSphere integration automatically provisioned a new node, and the old IP address now belongs to a new VM somewhere outside of Tanzu.

However, despite this situation occurring some days previously the Avi Kubernetes Operator (ako) has not cleaned up, perhaps expecting that the VM might be recovered eventually.

If you’re in a similar situation you will now know the name of the environment and should be able to determine the IP addresses of your current control plane nodes still:

kubectl config use-context [name of your management cluster context]
kubectl get nodes -o wide

In this case we are only interested in the IP addresses belonging to nodes having the control-plane node (the first three in the output below).

There aren’t any more ‘missing’ control plane endpoints shown above, so Kubernetes appears satisfied that it is in a workable state.

As a validation, check that the endpoints listed within the Kubernetes service map onto the current working list of nodes.

List the endpoints for the Kubernetes service (in default namespace)

kubectl get ep kubernetes -o json

The JSON output above is quite simple to read vertically, and confirms that there are three IP addresses within a subset of endpoints serving the Kubernetes API service on port 6443 (via the Avi Load Balancer vserver) that is defined in your ~/.kube/config file.

These match the output which the NSX Advanced Load Balancer showed previously.


What puzzled me for a very long time now seems obvious, that you cannot edit/remove any defunct entries from the Avi IP pool using the UI because the operator synchronises the list of endpoints for each service. By fixing the condition in Kubernetes the operator will take care of the content of the pool itself.

This is the way.

Obtain the list of services in the tkg-system namespace

kubectl get svc -n tkg-system

Now use the cluster-specific named control plane service to output the list of endpoints for the control plane

Aha, there’s the 172.20.11.45 control plane node which no longer exists in the cluster.

Edit the endpoint and manually remove the missing address from the subset addresses section

kubectl edit ep [tkg-system-tkg-mgmt-projit-control-plane] -n tkg-system

Using the VI editor remove the two lines declaring the ip and nodeName entries for the missing cluster node

Close the file and save the changes, the endpoint will be updated.

Refresh the Avi load balancer UI and if everything is well the pool will be updated dynamically when the ako operator detects the updated list of endpoints.


Further information confirming the status update is reflected in the ako-0 pod logs, which shows that a change has been detected between the cached copy of the virtual server object and the updated relationship which is computed from the graph database.

kubectl get logs ako-0 -n avi-system

It then resynchronises the pool content with Avi.

I’d be very pleased to hear if you run into a similar scenario, as I do not think that this element of ako is described anywhere in the official documentation of either Tanzu or AKO – and the DHCP lease re-issue will often crop up if an admin did not take care of making a permanent reservation after a node is added. Often this is because Tanzu will discover a broken node and intervene without anyone being aware of the problem, but this does not always make sense if addresses are not reserved permanently by default in your subnet.

Thanks for reading –

How much space does an air gap installation of Tanzu TKG 2.1.1 need?

In a follow up post to how-much-space-does-an-air-gap-installation-of-tanzu-tkg-1-6-0-need I thought it would be useful to expand on the initial summary to include an upgrade to TKG 2.1.1.

In the previous 1.6.0 example there was a total of 157 images (881 artifacts) requiring 9.7GB of storage space. However the download process has been modified and doesn’t use a shell script to download files for an air gap registry anymore, but rather a command such as:

tanzu isolated-cluster download-bundle --source-repo projects.registry.vmware.com/tkg --tkg-version v2.1.1

This results in 244 tar files being downloaded for a single version of TKG and 45GB of space needed.

When these tar files are uploaded I experienced several problems caused by a redis bug when using Harbor 1.10.x, and the upload command only succeeded once I had upgraded to Harbor 2.5.0.

tanzu isolated-cluster upload-bundle --source-directory ./ --destination-repo registry.sbcpureconsult.internal/tkg --ca-certificate /tmp/ca.crt

In total (for the combination of both TKG 1.6.0 and 2.1.1 releases) there are a total of 177 repositories requiring 20.58GB of storage space.

If I subtract the two figures from one another it indicates that TKG 2.1.1 requires 10.88GB in total.

How much space does an air gap installation of Tanzu TKG 1.6.0 need?

I have implemented several air gapped installations of Tanzu Kubernetes Grid 1.6 now using Harbor registry so thought it might be worth recording how many images are stored and the space required.

Example clean registry with only TKG 1.6 files

Short on time? I should caveat that my results only record the space needed for a single version of Kubernetes (1.23.8). This is the newest supported build of Kubernetes in the TLG 1.6.0 release.

During the air gap installation it is possible to reduce the file set required to be stored in your registry by extracting the Bill of Materials for a specific version only:

export DOWNLOAD_TKRS="v1.23.8_vmware.2-tkg.1"

In total (for this specific release) there are 157 images (881 artifacts) requiring 9.7GB of storage space.

I have tested the deployment of a management and worker cluster from the air gap registry and confirm successful installation.

Over time you may accumulate older versions in your registry which are no longer required, however there’s not information available currently on how you could reduce the number of images stored – so I would recommend keeping the image-copy file produced during each iteration of the air gap registry preparation phase so that you could remove them manually at a later date.

Comparing YAML documents with Beyond Compare

I’m a huge fan of Scooter Software’s magnificent Beyond Compare tool, having used it for years in many different scenarios. It does a brilliant job of lining up the similarities and differences in multiple file types – especially text based file formats.

A Kubernetes container orchestration project likely involves defining multiple YAML manifests for creating different objects in the API so I often spend many happy hours comparing differences between folders full of files in different environments.

Unfortunately within Beyond Compare there isn’t any inbuilt YAML file support so it sometimes makes mistakes when aligning functionally equivalent but structurally different files.

Here’s an example:

Default file comparison using simple everything else text format

By default red text shows differences, blue text means minor differences and hatched areas show where there is a lack of alignment possible. I have defined certain non-important grammar elements to ignore, e.g. secretName: so these are shown in blue, but why is it not possible to line up the other elements?

In discussion with Scooter Software they explained that they would need a YAML parsing algorithm to be developed in order to pre-sort such lists into the correct layout, and this isn’t possible right now.

Alternatively they suggested creating a custom File Type which uses an external program to do the sorting work before it attempts to line up and search for differences.

With that in mind I came across the yq project in Github by Mike Farah which provides amongst other things a sorting tool along the lines of jq (for JSON) for YAML documents instead. The following brief instructions show how to create a custom file type in Beyond Compare to have yq sort the keys and values in a temporary file before showing the comparison view. You can then choose to integrate and save changes on either side.

Follow preparation steps

  1. Download yq_windows_amd64.exe from the release page and rename the file to jq.exe for ease of use
  2. Create a folder called YAML in %USERPROFILE%\AppData\Roaming\Scooter Software\Beyond Compare 4\Helpers
  3. Copy the jq.exe file into the above folder
  4. Create a new file within the same folder yaml_sorted.bat containing:

    @echo off
    Helpers\YAML\yq.exe e -M "sort_keys(…)" %1 >>%2

  5. Open Beyond Compare and configure as follows

Configure Beyond Compare

Choose Tools, File Formats

Choose the [+] button and select Text Format

On the General tab enter a file mask to use when applying the File Format correctly, e.g. *.yaml,*.yml

Click on the Conversion tab and select External program (Unicode filenames) then enter the following path and parameters

Helpers\YAML\yaml_sorted.bat %s %t

Optionally, choose whether to Disable editing (when displaying the resulting comparison) and whether to ‘Trim trailing whitespace’ when saving – I find this helpful sometimes.

Click on the Misc tab and choose ‘Insert spaces instead of tabs’ and tab stop 8.

Click Save and Close.

Now find two YAML files to compare and double click in Beyond Compare. It will use the file type mask to apply the custom file format you just created.

In the background it will run the yq executable twice (via the batch file), once for each input file (left and right comparison). The executable takes the input file (%1) then sorts the keys and values alphabetically in sequence first before outputting the content to a new temporary file (%2).

Now open a YAML file comparison and you’ll find a much improved alignment based upon first the sorting of individual keys, then subsequent scalar values within each.

e.g. yq e -M -P "keys" C:\Users\stwa\desktop\example-ingress1.yaml

Here is the resulting comparison after processing with yq. I think you’ll agree that it’s much easier to spot the differences and also allow standardisation of your files going forward.

Resulting file comparison between left and right – after sorting via yq

There’s much more that you can do with both yq and Beyond Compare to further tune the formatting order, but hopefully this gives you some sort of light at the end of the tunnel when attempting comparisons of YAML documents.

I would like to investigate whether it’s possible to define a template format in yq so that it uses a custom key alignment order so that it matches the specification used by the Kubernetes API. In addition Beyond Compare allows line weighting within the algorithm so that it can assign preferential weights to certain grammar elements, although this has not been particularly fruitful when used according to the example below.

Final thoughts

Custom file types don’t just affect comparisons – If you right click on a file and choose Open With, Text Edit you will also see the sorted version of the file – not the original.

NB – If you save the modified content it will be saved using the new sorted format.

Documentation for yq is available here: https://mikefarah.gitbook.io/yq

NSX-T Manager appliance high-CPU whilst idle

I run VMware NSX-T in a small lab environment based on Intel NUCs, but I’ve noticed recently that even when not being challenged e.g. following initial boot and being essentially idle, the Manager appliance suffers continual high CPU usage which leads eventually to an uncomfortably warm office.

Even though running the correct minimum virtual machine hardware for the appliance has been configured, i.e. 4 vCPU and 16GB RAM, it was regularly using ~4.5GHz of physical CPU.

Here’s a good example of an otherwise idle appliance showing 40% CPU usage.

~40% CPU on a 4 vCPU virtual appliance

After connecting over SSH as the ‘admin’ user and entering ‘get process monitor’ it’s quickly apparent from the top output that ‘rngd’ is responsible for the majority of the CPU utilisation:

‘get process monitor’ whilst logged in as NSX-T admin console user

But what is this? A quick search of more general Linux resources informs us that it is a random number generator used in ensuring sufficient ‘entropy’ is available during creation of certificates, SSH keys etc.

In order to discover more about the purpose of this daemon we can inspect the description of the installed version (5-0ubuntu4nn1) under the current Ubuntu 18.04.4 LTS release.

apt show rng-tools/now

Description: Daemon to use a Hardware TRNG
The rngd daemon acts as a bridge between a Hardware TRNG (true random number
generator) such as the ones in some Intel/AMD/VIA chipsets, and the kernel’s
PRNG (pseudo-random number generator).
.
It tests the data received from the TRNG using the FIPS 140-2 (2002-10-10)
tests to verify that it is indeed random, and feeds the random data to the
kernel entropy pool.
.
This increases the bandwidth of the /dev/random device, from a source that
does not depend on outside activity. It may also improve the quality
(entropy) of the randomness of /dev/random.
.
A TRNG kernel module such as hw_random, or some other source of true
entropy that is accessible as a device or fifo, is required to use this
package.
.
This is an unofficial version of rng-tools which has been extensively
modified to add multithreading and a lot of new functionality.

So we know that this is a helper daemon which improves the speed of providing near-truly random numbers when applications ask for them. What version do we currently have installed in the NSX-T 3.1.2 manager appliance?

apt search rng-tools
Sorting... Done
Full Text Search... Done
rng-tools/now 5-0ubuntu4nn1 amd64 [installed,local]
  Daemon to use a Hardware TRNG

This appears to be the latest available version. In order to examine the status of the rngd daemon itself, log in to the appliance console as the root user and use:

systemctl list-units rng-tools.service

The service is shown as running,

Name of rngd random number generator service
root@nsx-manager:~# systemctl status rng-tools.service
rng-tools.service
Loaded: loaded (/etc/init.d/rng-tools; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-10-05 09:31:59 UTC; 17min ago
Docs: man:systemd-sysv-generator(8)
Process: 886 ExecStart=/etc/init.d/rng-tools start (code=exited, status=0/SUCCESS)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/rng-tools.service
`-934 /usr/sbin/rngd -r /dev/hwrng
Oct 05 09:31:59 nsx-manager systemd[1]: Starting rng-tools.service…
Oct 05 09:31:59 nsx-manager rng-tools[886]: Starting Hardware RNG entropy gatherer daemon: /etc/init.d/rng-tools: assigning /dev/hwrng to access rdrand on cpu
Oct 05 09:31:59 nsx-manager rng-tools[886]: crw-rw-rw- 1 root root 1, 8 Oct 5 09:31 /dev/random
Oct 05 09:31:59 nsx-manager rng-tools[886]: rngd.
Oct 05 09:31:59 nsx-manager systemd[1]: Started rng-tools.service.

What else can you find out about what it is doing in the background?

rngd -v

Two instances of ‘read error’ are output, followed by two further entropy sources being the Intel/AMG hardware random number generator and AES digital random number generator (RNDG). The ‘read error’ issue appears to be normal behaviour as the package attempts to read sources which don’t exist. Both of the displayed sources indicate that the CPU instruction set includes the necessary flags to tell the VM that it can access hardware random number generation.

Verbose output from rngd daemon

I must say, at this point it’s not clear whether NSX-T requires this service to be running permanently or whether it’s a component which Linux uses as a background service in order only to optimise the generation of a random number feed. It seems that stopping the service does appear to eventually cause problems in my lab – so please attempt the next section with CAUTION.

systemctl stop rng-tools.service

This leads to a significant reduction in CPU consumption and running temperature of my ESXi nodes.

CPU usage decreases after stopping rngd service

It may also be possible to disable the service permanently, but since I don’t have a full explanation of the purpose of this service from an NSX-T point of view I would stop short currently from doing this.

systemctl disable rng-tools.service

In the meantime I am hoping that I can get someone within the NSX-T development team to investigate these findings and provide some more permanent kind of workaround.

Further investigation

Further reading around the subject led me to find an issue has been reported on certain CPUs leading to activity spikes, https://github.com/nhorman/rng-tools/issues/136 and newer versions promise to fix this problem. The article mentioned suggests adding the -x jitter option to the start command but this is not available in the version installed in NSX-T.

RNGD_OPTS="-x jitter -r /dev/hwrng"

You can locate and edit the startup parameters by altering the service definition:

vi /etc/init.d/rng-tools

and potentially altering the default kernel values which are referenced by:

RNGDOPTIONS=
[ -r /etc/default/rng-tools ] && . /etc/default/rng-tools

Edit using:

vi /etc/default/rng-tools

However until the version of rng-tools used in NSX-T is updated to resolve this apparent issue it remains a personal choice as to whether or not the service can be stopped intermittently when a lab environment is not needed.


https://en.wikipedia.org/wiki/RDRAND
https://www.exoscale.com/syslog/random-numbers-generation-in-virtual-machines/

Citrix Gateway 13.0 Registry value EPA scan examples

If you’re having trouble with getting Citrix Endpoint Analysis scans of client device registry values to work properly (on Citrix Gateway) you may come across the following issue I experienced in the latest versions of firmware.

It appears that the EPA scan functionality in the NS 13.0 GUI (this article relates to 13.0.82.45) has been merged so that the numeric/non-numeric registry scan types now coalesce into one type of scan: REG_PATH; whereas in previous versions string values were interpreted using REG_NON_NUM_PATH.

Here’s a screen shot of the new expression editor drop down for Windows client EPA scans

NS13.0.82.45 drop down for Windows EPA scans

In comparison to the previous version (NS13.0.71.44).

NS13.0.71.44 drop down for Windows EPA scans

Here’s a screenshot of the registry scan entry panel where you can enter registry path and value, plus comparison or presence operators. Note the tooltip box which says that numeric comparisons will be done when using <,>,== etc.

NS13.0 registry scan value/comparison entry GUI

The convergence of these two types of scan into one appears to hide a reduction in comparison functionality, which only emerges once you attempt to use a string based registry value comparison using REG_PATH. You cannot use == anymore with string values such as REG_SZ.

This is a quick summary of the new behaviour following my own testing:

Numeric comparisons

Scans based upon REG_DWORD, REG_QWORD, REG_BINARY values will only work when carrying out boolean comparisons on numeric values with operators such as ==, !=, >=

e.g.

sys.client_expr("sys_0_REG_PATH_==HKEY\\_LOCAL\\_MACHINE\\\\SOFTWARE\\\\Classes\\\\YourRegistryKeyLocation\\\\YourRegistryValueName_VALUE==_12345[COMMENT: Registry]")

will result in a successful scan when YourRegistryValueName == 12345.

String comparisons

However when using the newly merged functionality, scans based upon REG_SZ values will only work when carrying out comparisons on string values using operators such as ‘contains’, ‘notcontains’.

If you try to use == as the operator on a string comparison the EPA scan logs will result in:

2021-09-28 09:25:38.883 Boolean compare failed. Value false operator ==
2021-09-28 09:25:38.883 Scan 'REG_PATH_==_HKEY\_LOCAL\_MACHINE\\SOFTWARE\\Classes\\YourRegistryKeyLocation\\YourRegistryValueName_VALUE_==_12345' failed for method 'VALUE'

Therefore modify your EPA action expression to fit the following example using ‘contains’:

sys.client_expr("sys_0_REG_PATH_==_HKEY\\\\_LOCAL\\\\_MACHINE\\\\\\\\SOFTWARE\\\\\\\\Classes\\\\\\\\YourRegistryKeyLocation\\\\\\\\YourRegistryValueName_VALUE_contains_12345[COMMENT: Registry]")

There are several other comparisons which do not appear to work properly, e.g. a numeric registry comparison of a REG_QWORD value which is longer than that allowed by the Citrix EPA plugin BUT is allowed within the 64 bytes of the Windows Registry value.

So my advice would be to consider whether the version of Citrix ADC you’re currently using actually offers the type of scan which you’re intending to use (REG_NON_NUM_PATH, REG_PATH), and NOT to rely upon documented examples without determining if the operator matches the value type correctly.

Further reading

https://support.citrix.com/article/CTX209148 – How to enable client EPA logging/troubleshooting

https://docs.citrix.com/en-us/citrix-gateway/current-release/vpn-user-config/advanced-endpoint-analysis-policies/advanced-endpoint-analysis-policy-expression-reference.html