I recently went to Prague to do my Automation Framework Master Class bringing my Portable Data Center as always. I did a similar class in Copenhagen in February and in both classes we saw Windows Module Installer High CPU utilization. Just that this time in Prague we got hit much harder.
This problem became a big headache for me, I actually canceled all future Master Classes. The dirty workaround we used in the classes was using GPO to disable the Windows Module Installer service, but that automatically disables the possibility to add Roles and Features! Time to resolve and share once and for all.
In the Master Classes we’re using Intel NUC and Skull with 32GB of memory running Citrix XenServer. The ONLY reason for using Citrix Hypervisor is the fact that there’s no requirement for a memory eating Appliance or Controller VM.
These small computers are super powerful and I’ve used them without any problems what so ever in Orlando, Amsterdam, London and Chicago in 2017. So something has changed between September 2017 and February 2018. Let’s dig in.
Suspect #1
Given the time frame above the primary suspect was the Spectre and Meltdown fix from Microsoft. Luckily that was not the case.
Suspect #2
1 |
Disable-ScheduledTask "\Microsoft\Windows\Servicing\StartComponentCleanup" |
What’s in common for all these Master Class training locations are slow, weak or non-existing internet connection. Even though all VMs used in the lab are fully patched and Windows Update is disabled I suspected that Microsoft had some wired stuff going on in the background. One of these is the Scheduled Task StartComponentCleanup located in \Microsoft\Windows\Servicing. Nobody knows exactly what it does and when it runs because there’s no triggers or actions.
“Windows 10 and Windows Server 2016 automatically reduce the size of the WinSxS folder by using methods similar to the ones described in this topic, in addition to internal processes, such as uninstalling and deleting packages with components that have been replaced by other components with newer versions. Previous versions of some components are kept on the system for a period of time, allowing you to rollback if necessary. After a period of time, these older components are automatically removed from the installation.”
Suspect #3
The graph below shows 5 VM’s with 2 vCPU and 2GB of memory.
The third suspect or scenario was that Windows Store Apps started to update themself after reboot. This has been covered by John in the post Fixing why Sysprep fails in Windows 10 due to Windows Store updates. This didn’t happen in my lab because of HideShell=YES but I tested disabling that Scheduled Task as well without any effect on the problem.
1 |
Disable-ScheduledTask "\Microsoft\Windows\WindowsUpdate\Automatic App Update" |
Suspect #4
The image below shows how the Windows Modules Installer going crazy for hours and days without ever finishing.
The fourth suspect where one of the many Maintenance tasks, so many that we have a one-liner to disable them all.
1 |
Get-ScheduledTask | ? {$_.Settings.MaintenanceSettings} | Disable-ScheduledTask |
Suspect #5
The graph below shows the CPU activity on one VM. Since I was now testing at home with fast internet that slow internet theory fell apart.
The fifth suspect was Microsoft Compatibility Appraiser which has been covered as a problem in this post.
Unfortunately that drives up CPU usage and brings computers (and much worse, VMs) to their knees
1 2 3 4 5 6 |
Disable-ScheduledTask "\Microsoft\Windows\Application Experience\AitAgent" Disable-ScheduledTask "\Microsoft\Windows\Application Experience\Microsoft Compatibility Appraiser" Disable-ScheduledTask "\Microsoft\Windows\Application Experience\ProgramDataUpdater" Disable-ScheduledTask "\Microsoft\Windows\Customer Experience Improvement Program\Consolidator" Disable-ScheduledTask "\Microsoft\Windows\Customer Experience Improvement Program\KernelCEIPTask" Disable-ScheduledTask "\Microsoft\Windows\Customer Experience Improvement Program\Use CEIP" |
Scenario #6
The graph below shows 7 VM’s with 2 vCPU and 2GB of memory.
As you might have figured out by now I’m about to go crazy and running out of options. I also tried to disable services like Windows Defender, Citrix XenServer Installation and Update Agent and the Scheduled Task Citrix Management Agent Auto-Updater without any luck.
The Guilty One
Then, for some reason I set the Citrix XenServer Windows Management Agent service to disabled, rebooted and BOOM the Windows Module Installer High CPU problem was gone.
I’ve been running XenServer Tools version 7.0.783 forever on XenServer 7.1, XenServer 7.1 CU1 and XenServer 7.4 without any problems, why now?
The image above is what Citrix calls silent install….In fact any version later than 7.0.783 will popup this message and it only goes away after 3 reboots. I’ve been bugging the XenServer team for years, but nobody seems to care, what a shame. But there might finally be a solution, let’s hope for the best.
Now since I’m running Infrastructure as Code I can simply change the XenServer tools version and spin up a complete new environment by Clicking One Button. The graph below shows 7 VM’s with 2 vCPU and 2GB of memory.
Once again Citrix Hypervisor creates big headaches, here’s another one for the XenServer team to look into Citrix XenServer – Slow Network Performance in WinPE Fixed
The Stranger Things
Yeah I know, the PM David Cottingham will blame me for not running the latest version. Tell you what, I actually have the same tool version on 2 other VMs on the same Hypervisor built from the same Reference Image working okay. GO FIGURE THAT OUT
Hope you enjoyed this article about how Windows Module Installer killed my CPU. How can you find out if you’re affected? Tools like ControlUp and uberAgent shows you the top CPU processes in their Dashboards.
Even if you don’t have this problem, the optimizations steps I preformed is perfect for non-persistent VMs and currently not covered by BISF or Citrix Optimizer.
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# Disable Maintenance Tasks Get-ScheduledTask | ? {$_.Settings.MaintenanceSettings} | Disable-ScheduledTask Disable-ScheduledTask "\Microsoft\Windows\Servicing\StartComponentCleanup" Disable-ScheduledTask "\Microsoft\Windows\WindowsUpdate\Automatic App Update" # Disable Microsoft Compatibility Appraiser # http://www.ampelofilosofies.gr/software/2015/07/10/forgetmenot-turn-off-windows-compatibility-appraiser Disable-ScheduledTask "\Microsoft\Windows\Application Experience\AitAgent" Disable-ScheduledTask "\Microsoft\Windows\Application Experience\Microsoft Compatibility Appraiser" Disable-ScheduledTask "\Microsoft\Windows\Application Experience\ProgramDataUpdater" Disable-ScheduledTask "\Microsoft\Windows\Customer Experience Improvement Program\Consolidator" Disable-ScheduledTask "\Microsoft\Windows\Customer Experience Improvement Program\KernelCEIPTask" Disable-ScheduledTask "\Microsoft\Windows\Customer Experience Improvement Program\Use CEIP" |
Great Article
We are running XenTools 7.1.1044 on XenServer 7.1 CU1 and are seeing the exact same issue.
Any news from XenServer team?
Nothing…but use this post to always install the lastest version – https://xenappblog.com/2018/download-and-install-latest-citrix-xenserver-tools/