Last two weeks I was engaged with some Nutanix cluster upgrades and I had some different issues in my clusters. So I decided to write this post to summarize the troubleshooting steps I have taken and if you are not confident running these commands and do the troubleshooting by yourself, always engage with Nutanix Support.
Before any upgrade it is always recommended to view the release notes for fixed and known bugs for particular version, check the Compatibility matrix and Upgrade Path as well. Here I have listed following important documents for you.
Software Documentation – for release notes for each software component.
Before any upgrade it is always recommended to view the release notes for fixed and known bugs for particular version, check the Compatibility matrix and Upgrade Path as well. Here I have listed following important documents for you.
Software Documentation – for release notes for each software component.
Upgrade path – to check your current software version can be upgraded to the desired or latest version. In this simple form you have to select the Software type, current version and target version. So it will show you whether it is supported or not by Green or Red colour respectively.
 |
| Nutanix upgrade paths |
Software Product Interoperability – From here you can check the Nutanix software component compatibility with each other. If your Prism Elements are registered with Prism Central it is always recommended to upgrade Prism Central first then each Prism Elements.
 |
| Software Product Interoperability |
Check Uptime and Last Reboot time for CVM or AHV Host
During my upgrades, I wanted to check the uptime in both CVMs and AHV Hosts because I wasn’t sure they were properly rebooted. So I used following steps to check the uptime of each component.
Check the uptime of CVM.
Login to CVM via SSH and the enter the following command. This will show the current CVM uptime. If you want to run this command cluster wide, use ‘allssh’ parameter first.
nutanix@cvm$ uptime
nutanix@cvm $ allssh uptime
This will display the current time, Up Time of each CVM, current user session and load average details.
 |
| uptime |
Check the last reboot details for CVM.
nutanix@cvm$ last reboot
nutanix@cvm$ allssh ‘last reboot’
 |
| last reboot |
This will display the last rebooted timestamp for each CVM.
Check the uptime of AHV hosts in the cluster.
Login to CVM via SSH and the enter the following command. This will show the current uptime of each host in the cluster.
nutanix@cvm$ hostssh uptime
 |
| uptime of AHV hosts |
Check the last reboot details for AHV hosts.
Login to CVM via SSH and the enter the following command. This will show the current last reboot timestamp of each host in the cluster.
nutanix@cvm$ hostssh ‘last reboot’
 |
| last reboot |
In any case you wanted to get the CVM IPs of you cluster use following command.
nutanix@cvm$ svmips
Check any pending/ongoing or current upgrade status.
When you are in upgrade process, if you want to check the upgrade status of your CVMs you can use the following command in any CVM of the cluster
nutanix@cvm$ upgrade_status
 |
| upgrade status |
this will display the running AOS version, cluster upgrade method and status of each CVM.
If you want to check the AHV host upgrade status, you can use the following commands.
nutanix@cvm$ host_upgrade_status
Also you can use NCC checks to see the upgrade status of your cluster
nutanix@cvm$ ncc health_checks system_checks cluster_active_upgrade_check
Pre-Upgrade checks failed with Pre-Upgrade check:test_cluster_status.
If you happen to get the above error during a pre-upgrade check, this could be due to one of following reason.
1. Failed to execute a cluster status
Run the "cluster status" command as below. If it executes, run pre-upgrade check again. If it fails, engage Nutanix Support.
nutanix@cvm$ cluster_status
2. No services are running on [ip]
Run the "cluster start" command as below on CVM (Controller VM) IP mentioned in the message. If it fails, engage Nutanix Support.
nutanix@cvm$ cluster_start
3. [Service Name] service is down on [ip]
Run the "cluster start" command as below on CVM IP mentioned in the message. If it fails, engage Nutanix Support.
nutanix@cvm$ cluster_start
Put AHV hosts into maintenance mode.
During an upgrade process you may want to manually put AHV host/s into maintenance mode. This can be accomplished by using following commands.
List available AHV hosts
List available AHV hosts
nutanix@cvm$ acli host.list
 |
| acli host.list |
this will display the AHV hosts along with schedulable status which means the maintenance mode status. If Schedulable status is true means host is not in maintenance mode and currently holds production workload. So if you want to put a host/s into maintenance mode you can use following command. This will automatically migrate the VMs running on host into available hosts in cluster.
nutanix@cvm$ acli host.enter_maintenance_mode <<IP of the Host>>
once the host is put into maintenance mode successfully the status will be shown in putty session. Also you can use the previous command to check the maintenance mode operation success or not.
 |
| AHV maintenance mode |
Exit AHV host/s from maintenance mode
To exit an AHV host/s from maintenance mode you can use the following command.
nutanix@cvm$ acli host.exit_maintenance_mode <<IP of the Host>>
Controller VM maintenance mode
If you want to put your CVM, you can use the following commands. As a parameter it required host ID which you can find using below command.
nutanix@cvm$ ncli host list
 |
| ncli host list |
once host ID identified use following command to enter CVM into maintenance mode.
nutanix@cvm$ ncli host edit id=<<host ID>> enable-maintenance-mode=true
use the same command to exit it from the maintenance mode with the switch ‘false’
nutanix@cvm$ ncli host edit id=<<host ID>> enable-maintenance-mode=false
Hope this commands will help you with your Nutanix upgrade. This is not a complete command line refference but I have inclueded some important commands in this post. If you find any other command/s useful during Nutanix upgrade process please comment them below.
Also following posts will be helpful for you if you have similar issues.
0 Comments