NSX-T | Migrating NVDS to VDS7.0 (trouble with vSAN timeout)

Are you experiencing errors when trying to migrate your NVDS to VDS7.0 in your VMware NSX-T 3.2.2 environment? You’re not alone, and we’re here to help. Migrating the NVDS through CLI commands can be a challenge, and some vSAN timeout errors can prevent successful migration. In this post, we’ll walk you through the errors we encountered during our own migration, and provide some solutions to help you make a successful migration. We’ll also provide some additional tips for renaming VDS and distributed port-groups, and removing opaque networks that may be blocking network deletion. Let’s dive in!

The migration process is carried out using CLI commands. However, I encountered a problem where the script only allowed 300 seconds to establish a connection with a host in Maintenance Mode. In certain cases, vSAN took longer than 5 minutes to empty the host, resulting in the script moving on to the next host without completing the migration. As a result, nothing was migrated in those cases. Let’s explore a few scenarios and discuss the solutions to these errors.

First of all, this is my production environmental:
– 18 hosts with vSphere 7.0 U3
– vCenter 7.0 U3
– NSX-T 3.2.2 (bonus: there is an upgrade to 3.2.3 to solve CVE-2023-20868, so please upgrade it!)
– Official Guide to migrate NVDS via CLI

After vds-migrate precheck and vds-migrate apply-topology commands, I used vds-migrate esxi-cluster-name <Cluster Name> to start migration of my NVDS to the new VDS7.0.

What went wrong? Initial host successfully migrated NVDS, while other hosts experienced errors and failed migration

The second and third hosts experienced delays while entering Maintenance Mode due to vSAN needing to resync certain objects. After 15 minutes, the script attempted to put the fourth host into Maintenance Mode. However, vSAN denied this action as two other hosts were already in Maintenance Mode. I encountered the same vSAN error with the remaining hosts until the script finally reached the end with:

NVDS Migrate Done:
1 Transport-Nodes Migrate Successfully, ID list ['<TN id>']
18 Transport-Nodes Migrate Failed, ID list <log list of TN information>
0 Transport-Nodes Migrate Skipped, ID list {}
More info see /var/log/nsx-cli/nsxcli.log

So, i checked /var/log/nsx-cli/nsxcli.log and I found info about first host migration and all other hosts timeout error:

2023-06-12T08:01:05.846Z 10176 cli.descriptors.cli_command_service INFO {10000} CMD: vds-migrate esxi-cluster-name <Cluster Name>
[first host]
2023-06-12T08:01:06.475Z 10176 cli.commands.manager.node_services INFO Putting Tn [6d57dc01-ef92-4083-xxx-xxxxxxxxxxxx] into VCMmode…
2023-06-12T08:04:43.259Z 10176 cli.commands.manager.node_services INFO Tn [6d57dc01-ef92-4083-xxx-xxxxxxxxxxxx] successfully entered VCMmode, start migrating…
2023-06-12T08:06:25.029Z 10176 cli.commands.manager.node_services INFO Tn [6d57dc01-ef92-4083-9xxx-xxxxxxxxxxxx] successfully migrated, exit VCMmode
[other hosts]
2023-06-12T08:06:43.222Z 10176 cli.commands.manager.node_services INFO Putting Tn [5d25c14b-cd17-4d9c-xxx-xxxxxxxxxxxx] into VCMmode…
2023-06-12T08:11:45.400Z 10176 cli.commands.manager.node_services WARNING Get VC Mmode timeout, tn id: 5d25c14b-cd17-4d9c-xxx-xxxxxxxxxxxx, expected: true
2023-06-12T08:11:45.492Z 10176 cli.commands.manager.node_services INFO Putting Tn [bedf4c15-a5c9-4a62-xxx-xxxxxxxxxxxx] into VCMmode…
2023-06-12T08:16:48.224Z 10176 cli.commands.manager.node_services WARNING Get VC Mmode timeout, tn id: bedf4c15-a5c9-4a62-xxx-xxxxxxxxxxxx, expected: true
2023-06-12T08:16:48.340Z 10176 cli.commands.manager.node_services INFO Putting Tn [936263ce-e76f-4ecf-xxx-xxxxxxxxxxxx] into VCMmode…
2023-06-12T08:21:50.623Z 10176 cli.commands.manager.node_services WARNING Get VC Mmode timeout, tn id: 936263ce-e76f-4ecf-xxx-xxxxxxxxxxxx, expected: true
2023-06-12T08:21:50.726Z 10176 cli.commands.manager.node_services INFO Putting Tn [8629072f-537b-482f-xxx-xxxxxxxxxxxx] into VCMmode…
2023-06-12T08:26:52.968Z 10176 cli.commands.manager.node_services WARNING Get VC Mmode timeout, tn id: 8629072f-537b-482f-xxx-xxxxxxxxxxxx, expected: true
2023-06-12T08:26:53.081Z 10176 cli.commands.manager.node_services INFO Putting Tn [c0f6e09c-2418-40ad-xxx-xxxxxxxxxxxx] into VCMmode…
2023-06-12T08:31:55.402Z 10176 cli.commands.manager.node_services WARNING Get VC Mmode timeout, tn id: c0f6e09c-2418-40ad-xxx-xxxxxxxxxxxx, expected: true

..and so on with all hosts.

How to solve the migration (manually)

Manually put a host into Maintenance Mode and execute the CLI command vds-migrate with the tn-list option. Repeat this step for each host to perform the NVDS migration correctly.

1. Create a file, for example, /opt/tnhost01 and include the ID of the host in the file.
2. Put the host into Maintenance Mode manually.
3. Once the host has reached Maintenance Mode, use the following command to initiate the migration: vds-migrate tn-list /opt/tnhost01
4. You can monitor the progress in vCenter, and the host will automatically exit Maintenance Mode after the migration, which typically takes about 100 seconds.
5. Repeat all the steps outlined above for each remaining host that needs to be migrated.

By following these steps, you can manually address the migration process for each host and ensure a successful NVDS migration.

images

Problem fixed Bonus: some opaque networks don’t disappear after migration.

As per VMware Migrate Host Switch to vSphere Distributed Switch guide:

NSX-T is no longer represented as an opaque network after migration. You may need to update your scripts to manage the migrated representation of the NSX-T hosts.

But wait! I still see 3 opaque networks on my vCenter!

The reason is some VM Templates that are using these netowrks blocked networks deletion. Now we need to fix it.

Renaming VDS and Distribuite port-group

Last but not least! now we can rename out VDS and Distribuited port-group with some cutiest names.