VMUG Advantage Community

Expand all | Collapse all

vCenter 7 high CPU usage

  • 1.  vCenter 7 high CPU usage

    Posted 04-10-2020 10:08 PM
    Edited by Mike Gittelman 04-10-2020 10:12 PM
    Upgraded vCenter and ESXi hosts from 6.7 to vSphere 7.  Everything works great, except I noticed that idle CPU usage is about 4-5X more than before for vCenter.  It used to be less than 1% CPU, or around 200MHZ.  Now it's about 1GHZ constantly.

    Tried going so far as to create a fresh instance of Vcenter with no hosts or any configuration attached to it and it does the same thing.

    According to top, the /usr/java/jre-vmware/bin/java process seems to use most of it.

    I went to far as to disable every single service and plugin (on the new install) and still usage is high.

    Has anyone else upgraded and seen this yet?

    Mike


  • 2.  RE: vCenter 7 high CPU usage

    Posted 04-14-2020 01:07 PM
    Edited by John Burleigh 04-14-2020 01:07 PM
    Yes, I am seeing the increase as well.  This is in my home lab.  The VCSA 7.0 is currently running (2) Cisco C220 M3s with 7 vms running.  The VCSA 6.7 is running (3) Cisco B200 M3 Blades, but the chassis is off.  I will power them on here in an hour and then check the results.



    ------------------------------
    John
    Consultant
    ITTSC LLC
    Vancouver WA
    ------------------------------



  • 3.  RE: vCenter 7 high CPU usage

    Posted 04-15-2020 01:22 PM
    Are you going to be able to get your disk usage down again to 36GB?  If so, did you just thin it or did you use other creative ways? Explain?

    ------------------------------
    Dion Rowney
    System Administrator
    U of S
    Saskatoon SK
    ------------------------------



  • 4.  RE: vCenter 7 high CPU usage

    Posted 04-16-2020 01:51 PM
    Hi

    same behavior here. fresh install of VCSA 7.0 currently running without any hosts connected to it.


    ------------------------------
    Markus Bangerter
    Engineer/Specialist
    Quickline AG
    Nidau
    ------------------------------



  • 5.  RE: vCenter 7 high CPU usage

    Posted 06-02-2020 03:22 PM
    Edited by Mike Gittelman 06-02-2020 03:22 PM
    Thanks all for confirming. Anyone seen any updates or found a fix on this high CPU issue?

    ------------------------------
    Mike Gittelman
    N/A
    Mike Gittelman
    San Francisco CA
    ------------------------------



  • 6.  RE: vCenter 7 high CPU usage

    Posted 06-03-2020 12:11 AM
    Noticed VCSA, in general, any version I tried, is slow with the default settings.   Changed them to 8 VCPU, 12GB for each VCSA.   The performance is better.   Here's how things look on my end for comparison.

    VCSA 7.0 10300



    root@vcsa03 [ ~ ]# cat /proc/meminfo |grep -Ei "inactive|active|free|total"
    MemTotal:       12284932 kB
    MemFree:          406256 kB
    Active:          9448884 kB
    Inactive:        1559128 kB
    Active(anon):    8700892 kB
    Inactive(anon):   188436 kB
    Active(file):     747992 kB
    Inactive(file):  1370692 kB
    SwapTotal:      27254776 kB
    SwapFree:       27136604 kB
    VmallocTotal:   34359738367 kB
    HugePages_Total:       0
    HugePages_Free:        0
    root@vcsa03 [ ~ ]#
    

    top - 19:57:33 up  1:10,  1 user,  load average: 0.78, 0.91, 1.15
    Tasks: 333 total,   1 running, 332 sleeping,   0 stopped,   0 zombie
    %Cpu0  :   9.3/2.3    12[|||||||||||                                                                                         ]
    %Cpu1  :   7.9/2.6    11[|||||||||||                                                                                         ]
    %Cpu2  :   7.6/2.0    10[||||||||||                                                                                          ]
    %Cpu3  :   5.4/2.0     7[|||||||                                                                                             ]
    %Cpu4  :   5.5/2.1     8[|||||||                                                                                             ]
    %Cpu5  :   5.9/2.7     9[|||||||||                                                                                           ]
    %Cpu6  :   4.9/1.7     7[|||||||                                                                                             ]
    %Cpu7  :   6.7/2.0     9[|||||||||                                                                                           ]
    GiB Mem : 82.1/11.7     [                                                                                                    ]
    GiB Swap:  0.5/26.0     [                                                                                                    ]
    
      PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND
     4932 vapiEnd+  20   0 4799.5m 505.7m  21.2   4.2   6:22.78 S /usr/java/jre-vmware/bin/vmware-vapi-endpoint.launcher -Xmx192m -XX:CompressedClassSpaceSize=96m -Xss256k -XX:ParallelG+
     1284 root      20   0 5527.5m  36.8m  16.7   0.3  12:06.84 S /usr/java/jre-vmware/bin/java -XX:+UseStringDeduplication -XX:+OptimizeStringConcat -Dcom.sun.management.jmxremote=true+
     3026 root      20   0 4683.6m 349.4m   3.1   2.9   2:49.56 S /usr/java/jre-vmware/bin/lookupsvc.launcher -Xmx80m -XX:CompressedClassSpaceSize=20m -Xss256k -XX:ParallelGCThreads=1 -+
     1412 vdtc      20   0  187.4m   5.1m   2.1   0.0   1:35.03 S /usr/lib/vmware-vdtc/vdtc
     2149 root      20   0 5195.2m 519.0m   1.8   4.3   5:08.13 S /usr/java/jre-vmware/bin/vmware-stsd.launcher -Xmx378m -XX:CompressedClassSpaceSize=160m -Xss256k -XX:ParallelGCThreads+
     2103 root      20   0   22.0g  66.4m   1.7   0.6   0:43.04 S /usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif
     9732 vpxd      20   0 4441.5m 242.8m   1.6   2.0   1:56.46 S /usr/lib/vmware-vpx/vpxd
    11998 vsan-he+  20   0 1230.3m 356.4m   1.5   3.0   2:02.66 S /usr/lib/vmware-vpx/vsan-health/bin/vsanvcmgmtd -c /usr/lib/vmware-vpx/vsan-health/VsanVcMgmtConfig.xml -u /etc/vmware-+
     3031 root      20   0  443.5m 127.9m   1.4   1.1   0:28.33 S /usr/bin/python /usr/lib/applmgmt/base/bin/vherdrunner /usr/lib/applmgmt/transport/bin/serve
     7093 root      20   0  145.2m  14.2m   1.4   0.1   0:52.07 S /usr/lib/vmware-vmon/vapi/vmon-vapi-provider -p 8900 -l info
    
    
    
    
    root@vcsa03 [ /var/log ]# ps -ef|grep -Ei 4932
    vapiEnd+  4932  2137  7 18:48 ?        00:06:36 /usr/java/jre-vmware/bin/vmware-vapi-endpoint.launcher -Xmx192m -XX:CompressedClassSpaceSize=96m -X
    root     65506 23464  0 20:12 pts/0    00:00:00 grep --color=auto -Ei 4932
    root@vcsa03 [ /var/log ]#
    root@vcsa03 [ /var/log ]#
    root@vcsa03 [ /var/log ]#
    root@vcsa03 [ /var/log ]# lsof -p 4932|grep log
    vmware-va 4932 vapiEndpoint  cwd       DIR             254,10     4096  131111 /storage/log/vmware/vmon
    vmware-va 4932 vapiEndpoint  mem       REG                8,3    12212 2235781 /usr/lib/vmware/common-jars/slf4j-log4j12-1.7.28.jar
    vmware-va 4932 vapiEndpoint  mem       REG                8,3    61829 2235644 /usr/lib/vmware/common-jars/commons-logging-1.2.jar
    vmware-va 4932 vapiEndpoint  mem       REG                8,3   489884 2235747 /usr/lib/vmware/common-jars/log4j-1.2.17.jar
    vmware-va 4932 vapiEndpoint    1w      REG             254,10       41  132517 /storage/log/vmware/vapi/endpoint/vmware-vapi-endpoint.stdout
    vmware-va 4932 vapiEndpoint    2w      REG             254,10       87  132518 /storage/log/vmware/vapi/endpoint/vmware-vapi-endpoint.stderr
    vmware-va 4932 vapiEndpoint    3w      REG             254,10   244893  132080 /storage/log/vmware/vapi/endpoint/vmware-vapi-endpoint-gc.log.0.current
    vmware-va 4932 vapiEndpoint   19r      REG                8,3   489884 2235747 /usr/lib/vmware/common-jars/log4j-1.2.17.jar
    vmware-va 4932 vapiEndpoint   24r      REG                8,3    61829 2235644 /usr/lib/vmware/common-jars/commons-logging-1.2.jar
    vmware-va 4932 vapiEndpoint   28r      REG                8,3    12212 2235781 /usr/lib/vmware/common-jars/slf4j-log4j12-1.7.28.jar
    vmware-va 4932 vapiEndpoint  122w      REG             254,10  1111927  132522 /storage/log/vmware/vapi/endpoint/endpoint.log
    vmware-va 4932 vapiEndpoint  125w      REG             254,10     1415  132818 /storage/log/vmware/vapi/endpoint/vcentershim.log
    vmware-va 4932 vapiEndpoint  126w      REG             254,10   100854  132819 /storage/log/vmware/vapi/endpoint/endpoint-access.log
    vmware-va 4932 vapiEndpoint  127w      REG             254,10     1130  132820 /storage/log/vmware/vapi/endpoint/vmodl2swagger.log
    vmware-va 4932 vapiEndpoint  128w      REG             254,10     1270  132821 /storage/log/vmware/vapi/endpoint/jetty.log
    root@vcsa03 [ /var/log ]#
    


    The VCSA 7 had no hosts connected to it though.  Just a base install.  Until AD works with VCSA 7, I've no plans to use it yet.  



    VCSA 6.7

    root@vcsa02 [ ~ ]# cat /proc/meminfo |grep -Ei "inactive|active|free|total"
    MemTotal:       12294724 kB
    MemFree:         2145184 kB
    Active:          8538360 kB
    Inactive:        1002340 kB
    Active(anon):    7597108 kB
    Inactive(anon):    65036 kB
    Active(file):     941252 kB
    Inactive(file):   937304 kB
    SwapTotal:      27254776 kB
    SwapFree:       27254776 kB
    VmallocTotal:   34359738367 kB
    HugePages_Total:       0
    HugePages_Free:        0
    root@vcsa02 [ ~ ]#
    

    top - 23:30:11 up 41 min,  1 user,  load average: 0.30, 0.21, 0.94
    Tasks: 318 total,   1 running, 317 sleeping,   0 stopped,   0 zombie
    %Cpu0  :   8.0/1.2     9[|||||||||                                                                                           ]
    %Cpu1  :  27.2/0.7    28[||||||||||||||||||||||||||||                                                                        ]
    %Cpu2  :   6.6/0.6     7[||||||||                                                                                            ]
    %Cpu3  :   9.9/1.3    11[|||||||||||                                                                                         ]
    %Cpu4  :   6.3/1.2     7[|||||||                                                                                             ]
    %Cpu5  :   9.7/0.6    10[|||||||||||                                                                                         ]
    %Cpu6  :  10.4/0.8    11[|||||||||||                                                                                         ]
    %Cpu7  :   8.9/1.6    11[|||||||||||                                                                                         ]
    GiB Mem : 70.6/11.7     [                                                                                                    ]
    GiB Swap:  0.0/26.0     [                                                                                                    ]
    
      PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND
     2358 vapiEnd+  20   0 4688.9m 432.7m  56.0   3.6   4:32.91 S /usr/java/jre-vmware/bin/vmware-vapi-endpoint.launcher -Xmx160m -XX:CompressedClassSpaceSize=96m -Xss256k -XX:ParallelGC+
     2254 root      20   0 3153.7m 352.7m   5.8   2.9   2:01.39 S /usr/java/jre-vmware/bin/vmware-analytics.launcher -Xmx131m -XX:CompressedClassSpaceSize=64m -Xss256k -XX:ParallelGCThre+
     1895 root      20   0 5261.9m 543.5m   5.6   4.5   3:51.21 S vmware-stsd -procname vmware-stsd -home /usr/java/jre-vmware -server -pidfile /var/log/vmware/sso/tcserver.pid -errfile +
     1535 root      20   0 1443.7m  30.0m   2.5   0.2   0:32.71 S /usr/lib/vmware-vmafd/sbin/vmafdd -s
     2391 root      20   0 1446.1m  24.6m   1.8   0.2   0:30.06 S /usr/lib/vmware-rhttpproxy/rhttpproxy -r /etc/vmware-rhttpproxy/config.xml -d /etc/vmware-rhttpproxy/endpoints.conf.d -f+
     6038 root      20   0 5288.8m 496.5m   1.7   4.1   3:08.40 S /usr/java/jre-vmware/bin/vmware-sps.launcher -Xmx504m -XX:CompressedClassSpaceSize=128m -Xss256k -XX:ParallelGCThreads=1+
     1618 root      20   0   22.1g  63.2m   1.6   0.5   0:22.92 S /usr/lib/vmware-vmdir/sbin/vmdird -s -l 0 -f /usr/lib/vmware-vmdir/share/config/vmdirschema.ldif
     4473 vpxd      20   0 4432.1m 223.4m   1.5   1.9   0:53.95 S /usr/lib/vmware-vpx/vpxd
     2589 root      20   0  466.8m 131.3m   1.4   1.1   0:31.85 S /usr/bin/python /usr/lib/applmgmt/base/bin/vherdrunner /usr/lib/applmgmt/transport/bin/serve
     2255 root      20   0  935.7m  12.0m   1.1   0.1   0:02.32 S /usr/lib/vmware-statsmonitor/statsMonitor /etc/vmware/statsmonitor/statsMonitor.xml
     1411 root      20   0  961.4m  10.9m   0.9   0.1   0:25.34 S /opt/likewise/sbin/lwregd --syslog
    
    
    
    
    root@vcsa02 [ ~ ]# ps -ef|grep -Ei 2358
    vapiEnd+  2358  2202  6 Jun02 ?        00:04:56 /usr/java/jre-vmware/bin/vmware-vapi-endpoint.launcher -Xmx160m -XX:CompressedClassSpaceSize=96m -Xss256k -XX:ParallelGCThreads=1 -Dlog4
    root     30735 20391  0 00:02 pts/0    00:00:00 grep --color=auto -Ei 2358
    root@vcsa02 [ ~ ]#
    root@vcsa02 [ ~ ]#
    root@vcsa02 [ ~ ]#
    root@vcsa02 [ ~ ]# lsof -p 2358|grep -Ei log
    vmware-va 2358 vapiEndpoint  cwd       DIR              252,3     4096 393245 /storage/log/vmware/vmon
    vmware-va 2358 vapiEndpoint  mem       REG                8,3    12212 271547 /usr/lib/vmware/common-jars/slf4j-log4j12-1.7.28.jar
    vmware-va 2358 vapiEndpoint  mem       REG                8,3    61829 271466 /usr/lib/vmware/common-jars/commons-logging-1.2.jar
    vmware-va 2358 vapiEndpoint  mem       REG                8,3   489884 271525 /usr/lib/vmware/common-jars/log4j-1.2.17.jar
    vmware-va 2358 vapiEndpoint    1w      REG              252,3       41 393708 /storage/log/vmware/vapi/endpoint/vmware-vapi-endpoint.stdout
    vmware-va 2358 vapiEndpoint    2w      REG              252,3       87 393796 /storage/log/vmware/vapi/endpoint/vmware-vapi-endpoint.stderr
    vmware-va 2358 vapiEndpoint    3w      REG              252,3   326432 393797 /storage/log/vmware/vapi/endpoint/vmware-vapi-endpoint-gc.log.0.current
    vmware-va 2358 vapiEndpoint   17r      REG                8,3   489884 271525 /usr/lib/vmware/common-jars/log4j-1.2.17.jar
    vmware-va 2358 vapiEndpoint   22r      REG                8,3    61829 271466 /usr/lib/vmware/common-jars/commons-logging-1.2.jar
    vmware-va 2358 vapiEndpoint   26r      REG                8,3    12212 271547 /usr/lib/vmware/common-jars/slf4j-log4j12-1.7.28.jar
    vmware-va 2358 vapiEndpoint  110w      REG              252,3   336756 393806 /storage/log/vmware/vapi/endpoint/endpoint.log
    vmware-va 2358 vapiEndpoint  113w      REG              252,3     1588 393808 /storage/log/vmware/vapi/endpoint/vcentershim.log
    vmware-va 2358 vapiEndpoint  114w      REG              252,3      920 393809 /storage/log/vmware/vapi/endpoint/vmodl2swagger.log
    root@vcsa02 [ ~ ]#
    
    
    
    


    Some notes:

    I changed VCPU from the defaults once I realized Java consumes most of the CPU cycles.  Java is execution core ( thread )  centric.

    Benefit of additional cores is that tasks don't wait in a queue for available VCPU resources.  Tasks get on the CPU quicker and get off quicker.

    Disk IO is important too.  Slow storage translates to I/O wait times.

    Noticed that the full GC isn't happening on VCSA 7.0 but minor GC's are happening more often.  VCSA 6.7 does a full GC every 4 minutes.  Maybe this is contributing to what you're seeing?

    Used top with a 10-second average, to get a better idea.  The output above uses the 10-second average for top.

    Thx,



    ------------------------------
    Tom
    ------------------------------



  • 7.  RE: vCenter 7 high CPU usage

    Posted 06-03-2020 12:36 AM
    Thanks Tom.  I'm not really worried about performance as it's certainly fast enough.  I have 8 cores allocated to it with the default 12GB memory.  I noticed a similar java process using up CPU constantly:

    109720 root 20 0 0.0t 0.0t 6.0 0.3 671:41.51 S /usr/java/jre-vmware/bin/java -XX:+UseStringDeduplication -XX:+OptimizeStringConcat -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=13033 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.host=localhost -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.manageme+

    Did increasing the cores reduce the CPU usage in MHZ, or just the %?

    ------------------------------
    Mike Gittelman
    N/A
    Mike Gittelman
    San Francisco CA
    ------------------------------



  • 8.  RE: vCenter 7 high CPU usage

    Posted 06-03-2020 11:43 PM

    Hmm, didn't capture the before and after.  I would be interested to see that as well.

    Let me see what I can do in the next few days to get that for comparison.

    Cheers,

    TK



    ------------------------------
    Tom
    ------------------------------



  • 9.  RE: vCenter 7 high CPU usage

    Posted 06-05-2020 09:22 PM

    8 VCPU, VCSA 7.0 




    2 VCPU, VCSA 7.0


    2 VCPU, VCSA 7.0
    This is the CPU over a day once I lowered it down to 2 VCPU.  Naturally, it averages more, which is to be expected.  But interestingly, the MHz is lower:



    With 2 VCPU assigned, VCSA took nearly an hour to load before I could login.  8 VCPU was very quick, maybe less than 5 minutes to load up.  Not sure what to think of the load time vs MHz yet.

    Thx,



    ------------------------------
    Tom
    ------------------------------