Debug School

Cover image for vSAN Support Logs
Suyash Sambhare
Suyash Sambhare

Posted on

vSAN Support Logs

#devops #vmware #vsan #esxi

How to collect vSAN support logs and upload to VMware

vSAN support logs are included in a standard ESXi support bundle in the form of vSAN trace. The vSAN support logs are automatically collected by assembling the hosts' ESXi support bundles.

Because vSAN is an object-network based storage system deployed across several ESXi hosts, while troubleshooting vSAN difficulties, the entire cluster logs, including vCenter, are necessary for a thorough examination of the problem.

If you have a 5-host cluster with a RAID1 policy, the three components of that item are spread over three servers in the cluster as determined by the vSAN. However, the issue may be encountered on a host that does not own any of the components or the object in question that requires troubleshooting.
If you have a 10-host cluster with a RAID5 or RAID6 policy, the components that comprise the object will be distributed across 4 or 6 hosts, depending on the RAID 5/6 strategy used.
So, as you can see from the examples above, if you do not have the entire cluster logs for analysis, Vmware Support will not have enough information to conduct an accurate/complete analysis.

The only times VMware Support does not need the complete cluster logs are when a disc failure is reported on a specific host or when the backtrace does not include vSAN, LSOM, or DOM. In many cases, merely the afflicted host bundle will suffice. When troubleshooting vSAN Performance issues, more vSAN Performance data is needed for analysis.

vSAN

How to collect vSAN performance service data that needs to be uploaded to VMware Support ticket (GS) to analyse/review performance matrix of your vSAN/VxRail clusters.

These must be provided when the following issues are identified:

  • High latency on vSAN performance charts
  • Latency experienced by virtual machines
  • Slow throughput
  • High outstanding I/O
  • Due to your organization security policy, you are unable to enable Customer Experience Improvement Program
  • vCenter Server is not connected to the internet to upload CEIP data to VMware
  • vSAN cluster performance deteriorates and requires VMware support to resolve.

This method collects stats for only the last two days. The collecting operation will generate a lot of queries. This operation could take a while to complete.

Steps:

  1. Verify if the performance problem is still present.
  2. Through the vSAN cluster > Configure > vSAN > Services > Performance Service > Edit, enable Verbose and Network Diagnostics mode.
  3. Give the system ten to fifteen minutes to operate.
  4. Gather all the cluster's host logs, including the vCenter logs.
  5. Turn off Network Diagnostics and Verbose modes.
  6. Send VMware the logs so they may be examined.

Since host logs wrap more quickly than vCenter logs, it is strongly advised that the cluster logs be gathered prior to the vCenter logs to make sure we have all the necessary logging from the hosts at the time of the occurrence. To make sure we get the performance data at the moment of the performance event for debugging performance issues, it is best to gather the Master/Leader host first. It is recommended to gather host logs in tiny batches—no more than five at a time—when working with big clusters with more than twenty servers and collecting them via vCenter to prevent log corruption.

If the problem can be reproduced on demand but is not an ongoing issue, take the following actions:

  1. Use the vSAN cluster to enable verbose and network diagnostics mode by going to Configure, vSAN, Services, Performance Service, and Edit.
  2. Confirm the performance problem again
  3. Give the system ten to fifteen minutes to operate.
  4. Gather logs from vCenter and all servers in the cluster that are running ESXi 5. Turn off Verbose and Network Diagnostics mode.
  5. Send VMware the logs so they may be examined.

Since the cluster's performance statistics are contained in the Master host's bundle, the logs from that server are needed if the performance problem is interfering with the log collection process.

Check the following to identify the cluster's Master host:

  • vCenter -> vSAN Cluster -> Health -> Performance Service -> vSAN -> Monitor -> Stats Master Election
  • All production vSAN deployments must have the vSAN Performance Service enabled for the necessary performance statistics to be available.
  • This aids in locating problems or bottlenecks at different vSAN stack tiers.
  • Large log files are produced by the verbose mode, and they are stored in the scratch partition and perf stat database object.
  • It can take a while for the ESXI logs gathering to complete the lengthy bundle collection.

It is highly recommended that the cluster logs are collected before vCenter logs to ensure we get all the required logging from the hosts at the time of the event as host logs tend to wrap faster than vCenter logs.
When dealing with large clusters 20+ hosts and collecting logs via vCenter it is best to collect the host logs in small batches, no more than 5 at a time to ensure the logs do not get corrupt.

Ref: https://knowledge.broadcom.com/external/article?legacyId=2072796

Top comments (0)