The Case of the “Ghost” ESXi Host in NSX

Resolving ESXi Trace Issues in Broadcom VMware NSX:
A Step-by-Step Guide

Introduction

VMware NSX provides a comprehensive platform for network virtualization, offering powerful tools to manage, configure, and automate your network infrastructure. However, like any complex system, users may occasionally run into issues that can disrupt their environment. One such issue is the presence of traces of removed ESXi hosts under transport zones. This article will walk you through the steps to resolve this problem, including how to sync your NSX Manager and eliminate these lingering traces.

Understanding the Issue: Traces of Removed ESXi Hosts in Transport Zones

In VMware NSX, transport zones are used to define network segmentation boundaries for virtual machines and hosts. Occasionally, when an ESXi host is removed from vCenter, it may still appear in NSX under these transport zones, even though it is no longer part of the cluster.

This can cause confusion or network inconsistencies, especially when trying to identify the active hosts in your environment. The issue arises due to stale entries in the NSX Manager’s index that were not properly cleaned up when the ESXi host was removed.

This is the alarm example on Dashboard:

• Feature: Comunication
• Event Type: Management Channel To Transport Node Down
• Severity: Medium
• Alarm State: Open
• Description: Management channel to Transport Node <NodeName and IP Address> is down for 5 minutes.
• Recommended Action: Ensure there is network connectivity between the Manager nodes and Transport node <NodeName and IP Address> and no tirewalls are blocking tratic between the nodes.

Issue Verification: Confirming the Problem

In this particular case we identified that traces of the removed ESXi host still appeared under the transport zones in NSX. Importantly, the ESXi host was not listed under the vCenter cluster, other hosts, or as a standalone host, confirming that the removal process in vCenter was successful.

However, the host’s ghost entries persisted in NSX, leading us to the conclusion that the issue was related to NSX’s internal indexes, which had not been updated to reflect the removal of the ESXi host.

Cause Identification: Why Does This Happen?

The root cause of this issue lies in the way NSX manages host data. When an ESXi host is removed from vCenter, NSX doesn’t automatically delete the associated data from its internal indexes. As a result, the host remains listed under transport zones, causing confusion.

This happens because NSX relies on periodic synchronization to keep its internal records up to date with vCenter. If synchronization doesn’t occur properly, stale or outdated entries remain visible in the system.

• Investigation revealed that the removed ESXi host was not listed under “Cluster,” “Other,” or “Standalone” within the NSX UI.
• This indicated an inconsistency in the NSX database where the removed host’s entries were not properly removed upon its deletion from vCenter.

Solution Recommendation: Resyncing the Manager’s Indexes

To resolve this issue, the solution is straightforward: you need to resync the NSX Manager’s indexes. This process forces NSX to re-check and update its internal records, removing any outdated or removed entries, including the ESXi host that should no longer be visible.

Here’s a step-by-step guide to resync the indexes:

1. Access the NSX Manager UI: Log in to the NSX Manager using your credentials.
2. Navigate to the “System” Section: Once in the NSX Manager, go to the “System” tab to manage the NSX Manager settings.
3. Initiate the Resynchronization: Find the option to resync the indexes of the NSX Manager. This can usually be done under the “Health” or “Settings” section of the NSX UI.
4. Verify the Synchronization: After resyncing, check the transport zones again to ensure that the removed ESXi host no longer appears.
5. Force Remove the ESXi Host (if necessary): If the host still appears in the transport zones, you may need to force remove it from NSX by manually removing the entry.

Solution Justification: Why This Works

The process of resynchronizing the NSX Manager’s indexes effectively updates the records in NSX and clears out any remnants of removed hosts. This approach is supported by VMware’s official knowledge base article [KB: 319975], which provides further details on resolving similar issues.

By resyncing the indexes, you ensure that NSX accurately reflects the current state of your infrastructure and that no ghost entries from previously removed ESXi hosts remain in the system.

Conclusion

Issues like traces of removed ESXi hosts under transport zones can be frustrating, but they are relatively easy to resolve. By resyncing the NSX Manager’s indexes, you can quickly remove stale data and ensure that your network virtualization environment is accurate and up-to-date.

As VMware NSX continues to evolve, staying on top of routine maintenance tasks like index synchronization is crucial for ensuring smooth operations and troubleshooting any anomalies. For more information, consult Braodcom Knowledge Base or reach out to support for assistance.

---

By following these simple steps, you can keep your VMware NSX environment clean, consistent, and free of any unwanted traces of removed ESXi hosts, leading to a more efficient and streamlined network.