Data Center & Cloud Infrastructure VMUG Community

Expand all | Collapse all

VCENTER 7 HIGH CPU USAGE

  • 1.  VCENTER 7 HIGH CPU USAGE

    Posted 06-26-2020 02:39 PM
    #vSphereAMA 

    Reposting from here since I never figured this out:  https://community.vmug.com/communities/community-home/digestviewer/viewthread?GroupId=2779&MessageKey=aac7f06a-15ab-4eae-9a48-5a28770833b7&CommunityKey=5d966ad2-2bc7-4f01-8bf5-f20dbb092db9&tab=digestviewer&ReturnUrl=%2fcommunities%2fcommunity-home%2fdigestviewer%3fcommunitykey%3d5d966ad2-2bc7-4f01-8bf5-f20dbb092db9%26tab%3ddigestviewer

    Upgraded vCenter and ESXi hosts from 6.7 to 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ most of the time.  Now it's about 1GHZ constantly.


    Tried going so far as to create a fresh instance of vCenter 7 with no hosts or any configuration attached to it and it does the same thing.

    According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

    I went to far as to disable every single service and plugin (on the new install) and still usage is high.

    Any idea what's going on?


  • 2.  RE: VCENTER 7 HIGH CPU USAGE

    Posted 06-26-2020 03:12 PM
    Hello Mike,

    Can you take SSH to the vCenter and run the below command and send the output.

    cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr

    NOTE: Please do scrub any confidential data since you are posing it in public forum.

    ------------------------------
    Install/Upgrade Expert - Jonathan Ebenezer
    ------------------------------



  • 3.  RE: VCENTER 7 HIGH CPU USAGE

    Posted 06-26-2020 03:15 PM
    Must be missing something:

    root@vcenter [ ~ ]# cat vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr
    cat: vpxd-profiler.log: No such file or directory

    Is there a path I should specify for the log file?

    ------------------------------
    Mike Gittelman
    N/A
    Mike Gittelman
    San Francisco CA
    ------------------------------



  • 4.  RE: VCENTER 7 HIGH CPU USAGE

    Posted 06-26-2020 03:24 PM
    Ok, tried this:

    root@vcenter [ /var/log/vmware ]# cat /storage/log/vmware/vpxd/vpxd-profiler.log | grep -iE 'Session.ID' | cut -d "/" -f 4-7 | sort -u | cut -d "/" -f 3,4 | sort | uniq -c | sort -nr
    9 Username='VSPHERE.LOCAL\vpxd-extension-8482b45a-12ac-4c9b-8d1b-29823c1d5fe1'/ClientIP='127.0.0.1'
    5 Username='VSPHERE.LOCAL\vpxd-extension-8482b45a-12ac-4c9b-8d1b-29823c1d5fe1'/ClientIP='192.168.10.9'
    2 Username='VirtualCenter'/ClientIP='127.0.0.1'
    2 Username='com.vmware.vim.eam'/ClientIP='127.0.0.1'
    2 Username='AD\xxx'/ClientIP='192.168.10.9'
    1 Username='VSPHERE.LOCAL\vsphere-webclient-8482b45a-12ac-4c9b-8d1b-29823c1d5fe1'/ClientIP='192.168.10.9'
    1 Username='com.vmware.vsan.health'/ClientIP='127.0.0.1'
    1 Username='com.vmware.vim.vsm'/ClientIP='127.0.0.1'
    1 Username='com.vmware.vcIntegrity'/ClientIP='127.0.0.1'
    1 Username='AD\xxx'/ClientIP='192.168.10.146'

    Names of logins were removed.

    ------------------------------
    Mike Gittelman
    N/A
    Mike Gittelman
    San Francisco CA
    ------------------------------



  • 5.  RE: VCENTER 7 HIGH CPU USAGE

    Posted 06-26-2020 03:37 PM
    Hi,

    You got it right. Usually if we see a high number of sessions mostly from monitoring servers which will bombard requests to VC creating high CPU.
    No anomaly I can find from the output you shared.

    Can you run vim-top or just top on the VC and get the snippet.
    What is the load average that you are seeing there ?


    ------------------------------
    Install/Upgrade Expert - Jonathan Ebenezer
    ------------------------------



  • 6.  RE: VCENTER 7 HIGH CPU USAGE

    Posted 06-26-2020 03:50 PM



    ------------------------------
    Mike Gittelman
    N/A
    Mike Gittelman
    San Francisco CA
    ------------------------------



  • 7.  RE: VCENTER 7 HIGH CPU USAGE

    Posted 06-26-2020 04:24 PM
    Hi Mike,

    We do not find anything abnormal with your vCenter server.
    The idle CPU to 1GHz could be due to various new services introduced in vCenter 7.0.

    ------------------------------
    Install/Upgrade Expert - Jonathan Ebenezer
    ------------------------------



  • 8.  RE: VCENTER 7 HIGH CPU USAGE

    Posted 06-26-2020 04:46 PM
    Hi guys,

    I know this might not be the answer you are looking for, but definitely ensure best practices are being followed when it comes to running your vSphere 7 environment.

    Here are just a couple:

    https://blogs.vmware.com/performance/2020/05/performance-best-practices-guide-for-vsphere-7-0.html
    https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/vsphere-esxi-vcenter-server-70-performance-best-practices.pdf

    Good luck!

    ------------------------------
    Marvin Marcos
    Engineer/Specialist
    VMware
    Oakville ON
    ------------------------------