VMware vCenter has “no healthy upstream” error

Over the weekend, I was configuring our new VMware servers. I was happily working around when all of a sudden, vCenter started showing the hated “no healthy upstream” message on the vCenter website.

Thankfully, this was not the first time I’d seen this happen, and it usually occurs randomly (at least in my experience). The solution is easier than most people would think.

The first step is to log into vami portal. This is at the same FQDN as your vCenter server but on port 5480. So if your vCenter URL is https://vcenter.mycompany.com, the URL for your vami portal is https://vcenter.mycompany.com:5480. To log into the vami portal, you’ll need the password for your administrator@vshere.local account, which was set up when you first provisioned your VMware environment.

After you log in with that account, click on Networking on the menu on the left. From here, you can edit the networking settings for the vSphere server itself. Click the “Edit” link on the top right. When the page opens, select the NIC (there is probably only one) and click next.

The next page is where things got a little weird for me, as the entire middle section of the page was black. (No, I didn’t change anything when taking this screenshot.)

Thankfully clicking Next past this page and through the other pages of the wizard worked. When the wizard is done, the VM will restart, and about 10 minutes later, the problem (should) go away. (Give vCenter plenty of time to start, as vCenter is horribly slow at starting up.)

One other change I made to resolve the error was to increase the amount of RAM the VM had access to. When I first logged into the vami portal, it showed that the VM was running out of RAM, and I figured that since it was already down, now would be a great time to add memory (and our new servers give us plenty of RAM to use) so I pushed our vCenter server to 32 Gigs of RAM.

I’m not sure if this helped, but it was a change I made. As I didn’t have access to vCenter at this point, I had to connect to each host’s console until I found the host which was running the vCenter server so that I could edit the amount of RAM directly on the server.

Denny