Data Center & Cloud Infrastructure VMUG Community

#vSphereAMA 

Reposting from here since I never figured this out:  https://community.vmug.com/communities/community-home/digestviewer/viewthread?GroupId=2779&MessageKey=aac7f06a-15ab-4eae-9a48-5a28770833b7&CommunityKey=5d966ad2-2bc7-4f01-8bf5-f20dbb092db9&tab=digestviewer&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3d5d966ad2-2bc7-4f01-8bf5-f20dbb092db9%26tab%3ddigestviewer

Upgraded vCenter and ESXi hosts from 6.7 to 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ most of the time.  Now it's about 1GHZ constantly.

Tried going so far as to create a fresh instance of vCenter 7 with no hosts or any configuration attached to it and it does the same thing.

According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

I went to far as to disable every single service and plugin (on the new install) and still usage is high.

Any idea what's going on?

Hello Mike,

Can you take SSH to the vCenter and run the below command and send the output.

cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr

NOTE: Please do scrub any confidential data since you are posing it in public forum.

------------------------------
Install/Upgrade Expert - Jonathan Ebenezer
------------------------------

Original Message:
Sent: 06-26-2020 02:39 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

#vSphereAMA 

Reposting from here since I never figured this out:  https://community.vmug.com/communities/community-home/digestviewer/viewthread?GroupId=2779&MessageKey=aac7f06a-15ab-4eae-9a48-5a28770833b7&CommunityKey=5d966ad2-2bc7-4f01-8bf5-f20dbb092db9&tab=digestviewer&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3d5d966ad2-2bc7-4f01-8bf5-f20dbb092db9%26tab%3ddigestviewer

Upgraded vCenter and ESXi hosts from 6.7 to 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ most of the time.  Now it's about 1GHZ constantly.

Tried going so far as to create a fresh instance of vCenter 7 with no hosts or any configuration attached to it and it does the same thing.

According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

I went to far as to disable every single service and plugin (on the new install) and still usage is high.

Any idea what's going on?

Must be missing something:

root@vcenter [ ~ ]# cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr
cat: vpxd-profiler.log: No such file or directory

Is there a path I should specify for the log file?

------------------------------
Mike Gittelman
N/A
Mike Gittelman
San Francisco CA
------------------------------

Original Message:
Sent: 06-26-2020 03:12 PM
From: Install/Upgrade Expert - Jonathan Ebenezer
Subject: VCENTER 7 HIGH CPU USAGE

Hello Mike,

Can you take SSH to the vCenter and run the below command and send the output.

cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr

NOTE: Please do scrub any confidential data since you are posing it in public forum.

------------------------------
Install/Upgrade Expert - Jonathan Ebenezer

Original Message:
Sent: 06-26-2020 02:39 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

#vSphereAMA 

Reposting from here since I never figured this out:  https://community.vmug.com/communities/community-home/digestviewer/viewthread?GroupId=2779&MessageKey=aac7f06a-15ab-4eae-9a48-5a28770833b7&CommunityKey=5d966ad2-2bc7-4f01-8bf5-f20dbb092db9&tab=digestviewer&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3d5d966ad2-2bc7-4f01-8bf5-f20dbb092db9%26tab%3ddigestviewer

Upgraded vCenter and ESXi hosts from 6.7 to 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ most of the time.  Now it's about 1GHZ constantly.

Tried going so far as to create a fresh instance of vCenter 7 with no hosts or any configuration attached to it and it does the same thing.

According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

I went to far as to disable every single service and plugin (on the new install) and still usage is high.

Any idea what's going on?

Ok, tried this:

root@vcenter [ /var/log/vmware ]# cat /storage/log/vmware/vpxd/vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr
9 Username='VSPHERE.LOCAL\vpxd-extension-8482b45a-12ac-4c9b-8d1b-29823c1d5fe1'/ClientIP='127.0.0.1'
5 Username='VSPHERE.LOCAL\vpxd-extension-8482b45a-12ac-4c9b-8d1b-29823c1d5fe1'/ClientIP='192.168.10.9'
2 Username='VirtualCenter'/ClientIP='127.0.0.1'
2 Username='com.vmware.vim.eam'/ClientIP='127.0.0.1'
2 Username='AD\xxx'/ClientIP='192.168.10.9'
1 Username='VSPHERE.LOCAL\vsphere-webclient-8482b45a-12ac-4c9b-8d1b-29823c1d5fe1'/ClientIP='192.168.10.9'
1 Username='com.vmware.vsan.health'/ClientIP='127.0.0.1'
1 Username='com.vmware.vim.vsm'/ClientIP='127.0.0.1'
1 Username='com.vmware.vcIntegrity'/ClientIP='127.0.0.1'
1 Username='AD\xxx'/ClientIP='192.168.10.146'

Names of logins were removed.

------------------------------
Mike Gittelman
N/A
Mike Gittelman
San Francisco CA
------------------------------

Original Message:
Sent: 06-26-2020 03:12 PM
From: Install/Upgrade Expert - Jonathan Ebenezer
Subject: VCENTER 7 HIGH CPU USAGE

Hello Mike,

Can you take SSH to the vCenter and run the below command and send the output.

cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr

NOTE: Please do scrub any confidential data since you are posing it in public forum.

------------------------------
Install/Upgrade Expert - Jonathan Ebenezer

Original Message:
Sent: 06-26-2020 02:39 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

#vSphereAMA 

Reposting from here since I never figured this out:  https://community.vmug.com/communities/community-home/digestviewer/viewthread?GroupId=2779&MessageKey=aac7f06a-15ab-4eae-9a48-5a28770833b7&CommunityKey=5d966ad2-2bc7-4f01-8bf5-f20dbb092db9&tab=digestviewer&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3d5d966ad2-2bc7-4f01-8bf5-f20dbb092db9%26tab%3ddigestviewer

Upgraded vCenter and ESXi hosts from 6.7 to 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ most of the time.  Now it's about 1GHZ constantly.

Tried going so far as to create a fresh instance of vCenter 7 with no hosts or any configuration attached to it and it does the same thing.

According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

I went to far as to disable every single service and plugin (on the new install) and still usage is high.

Any idea what's going on?

Hi,

You got it right. Usually if we see a high number of sessions mostly from monitoring servers which will bombard requests to VC creating high CPU.
No anomaly I can find from the output you shared.

Can you run vim-top or just top on the VC and get the snippet.
What is the load average that you are seeing there ?


------------------------------
Install/Upgrade Expert - Jonathan Ebenezer
------------------------------

Original Message:
Sent: 06-26-2020 03:23 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

Ok, tried this:

root@vcenter [ /var/log/vmware ]# cat /storage/log/vmware/vpxd/vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr
9 Username='VSPHERE.LOCAL\vpxd-extension-8482b45a-12ac-4c9b-8d1b-29823c1d5fe1'/ClientIP='127.0.0.1'
5 Username='VSPHERE.LOCAL\vpxd-extension-8482b45a-12ac-4c9b-8d1b-29823c1d5fe1'/ClientIP='192.168.10.9'
2 Username='VirtualCenter'/ClientIP='127.0.0.1'
2 Username='com.vmware.vim.eam'/ClientIP='127.0.0.1'
2 Username='AD\xxx'/ClientIP='192.168.10.9'
1 Username='VSPHERE.LOCAL\vsphere-webclient-8482b45a-12ac-4c9b-8d1b-29823c1d5fe1'/ClientIP='192.168.10.9'
1 Username='com.vmware.vsan.health'/ClientIP='127.0.0.1'
1 Username='com.vmware.vim.vsm'/ClientIP='127.0.0.1'
1 Username='com.vmware.vcIntegrity'/ClientIP='127.0.0.1'
1 Username='AD\xxx'/ClientIP='192.168.10.146'

Names of logins were removed.

------------------------------
Mike Gittelman
N/A
Mike Gittelman
San Francisco CA

Original Message:
Sent: 06-26-2020 03:12 PM
From: Install/Upgrade Expert - Jonathan Ebenezer
Subject: VCENTER 7 HIGH CPU USAGE

Hello Mike,

Can you take SSH to the vCenter and run the below command and send the output.

cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr

NOTE: Please do scrub any confidential data since you are posing it in public forum.

------------------------------
Install/Upgrade Expert - Jonathan Ebenezer

Original Message:
Sent: 06-26-2020 02:39 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

#vSphereAMA 

Reposting from here since I never figured this out:  https://community.vmug.com/communities/community-home/digestviewer/viewthread?GroupId=2779&MessageKey=aac7f06a-15ab-4eae-9a48-5a28770833b7&CommunityKey=5d966ad2-2bc7-4f01-8bf5-f20dbb092db9&tab=digestviewer&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3d5d966ad2-2bc7-4f01-8bf5-f20dbb092db9%26tab%3ddigestviewer

Upgraded vCenter and ESXi hosts from 6.7 to 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ most of the time.  Now it's about 1GHZ constantly.

Tried going so far as to create a fresh instance of vCenter 7 with no hosts or any configuration attached to it and it does the same thing.

According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

I went to far as to disable every single service and plugin (on the new install) and still usage is high.

Any idea what's going on?

------------------------------
Mike Gittelman
N/A
Mike Gittelman
San Francisco CA
------------------------------

Original Message:
Sent: 06-26-2020 03:12 PM
From: Install/Upgrade Expert - Jonathan Ebenezer
Subject: VCENTER 7 HIGH CPU USAGE

Hello Mike,

Can you take SSH to the vCenter and run the below command and send the output.

cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr

NOTE: Please do scrub any confidential data since you are posing it in public forum.

------------------------------
Install/Upgrade Expert - Jonathan Ebenezer

Original Message:
Sent: 06-26-2020 02:39 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

#vSphereAMA 

Reposting from here since I never figured this out:  https://community.vmug.com/communities/community-home/digestviewer/viewthread?GroupId=2779&MessageKey=aac7f06a-15ab-4eae-9a48-5a28770833b7&CommunityKey=5d966ad2-2bc7-4f01-8bf5-f20dbb092db9&tab=digestviewer&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3d5d966ad2-2bc7-4f01-8bf5-f20dbb092db9%26tab%3ddigestviewer

Upgraded vCenter and ESXi hosts from 6.7 to 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ most of the time.  Now it's about 1GHZ constantly.

Tried going so far as to create a fresh instance of vCenter 7 with no hosts or any configuration attached to it and it does the same thing.

According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

I went to far as to disable every single service and plugin (on the new install) and still usage is high.

Any idea what's going on?

Hi Mike,

We do not find anything abnormal with your vCenter server.
The idle CPU to 1GHz could be due to various new services introduced in vCenter 7.0.

------------------------------
Install/Upgrade Expert - Jonathan Ebenezer
------------------------------

Original Message:
Sent: 06-26-2020 03:49 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

------------------------------
Mike Gittelman
N/A
Mike Gittelman
San Francisco CA

Original Message:
Sent: 06-26-2020 03:12 PM
From: Install/Upgrade Expert - Jonathan Ebenezer
Subject: VCENTER 7 HIGH CPU USAGE

Hello Mike,

Can you take SSH to the vCenter and run the below command and send the output.

cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr

NOTE: Please do scrub any confidential data since you are posing it in public forum.

------------------------------
Install/Upgrade Expert - Jonathan Ebenezer

Original Message:
Sent: 06-26-2020 02:39 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

#vSphereAMA 

Reposting from here since I never figured this out:  https://community.vmug.com/communities/community-home/digestviewer/viewthread?GroupId=2779&MessageKey=aac7f06a-15ab-4eae-9a48-5a28770833b7&CommunityKey=5d966ad2-2bc7-4f01-8bf5-f20dbb092db9&tab=digestviewer&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3d5d966ad2-2bc7-4f01-8bf5-f20dbb092db9%26tab%3ddigestviewer

Upgraded vCenter and ESXi hosts from 6.7 to 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ most of the time.  Now it's about 1GHZ constantly.

Tried going so far as to create a fresh instance of vCenter 7 with no hosts or any configuration attached to it and it does the same thing.

According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

I went to far as to disable every single service and plugin (on the new install) and still usage is high.

Any idea what's going on?

Hi guys,

I know this might not be the answer you are looking for, but definitely ensure best practices are being followed when it comes to running your vSphere 7 environment.

Here are just a couple:

https://blogs.vmware.com/performance/2020/05/performance-best-practices-guide-for-vsphere-7-0.html
https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere-esxi-vcenter-server-70-performance-best-practices.pdf

Good luck!

------------------------------
Marvin Marcos
Engineer/Specialist
VMware
Oakville ON
------------------------------

Original Message:
Sent: 06-26-2020 04:23 PM
From: Install/Upgrade Expert - Jonathan Ebenezer
Subject: VCENTER 7 HIGH CPU USAGE

Hi Mike,

We do not find anything abnormal with your vCenter server.
The idle CPU to 1GHz could be due to various new services introduced in vCenter 7.0.

------------------------------
Install/Upgrade Expert - Jonathan Ebenezer

Original Message:
Sent: 06-26-2020 03:49 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

------------------------------
Mike Gittelman
N/A
Mike Gittelman
San Francisco CA

Original Message:
Sent: 06-26-2020 03:12 PM
From: Install/Upgrade Expert - Jonathan Ebenezer
Subject: VCENTER 7 HIGH CPU USAGE

Hello Mike,

Can you take SSH to the vCenter and run the below command and send the output.

cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr

NOTE: Please do scrub any confidential data since you are posing it in public forum.

------------------------------
Install/Upgrade Expert - Jonathan Ebenezer

Original Message:
Sent: 06-26-2020 02:39 PM
From: Mike Gittelman
Subject: VCENTER 7 HIGH CPU USAGE

#vSphereAMA 

Reposting from here since I never figured this out:  https://community.vmug.com/communities/community-home/digestviewer/viewthread?GroupId=2779&MessageKey=aac7f06a-15ab-4eae-9a48-5a28770833b7&CommunityKey=5d966ad2-2bc7-4f01-8bf5-f20dbb092db9&tab=digestviewer&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3d5d966ad2-2bc7-4f01-8bf5-f20dbb092db9%26tab%3ddigestviewer

Upgraded vCenter and ESXi hosts from 6.7 to 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ most of the time.  Now it's about 1GHZ constantly.

Tried going so far as to create a fresh instance of vCenter 7 with no hosts or any configuration attached to it and it does the same thing.

According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

I went to far as to disable every single service and plugin (on the new install) and still usage is high.

Any idea what's going on?