I have a pretty new Windows 11 Pro installation, it has about a week. I noticed that my RAM usage is constantly going up. It was concerned because procexp readouts did not match the total used RAM value that's reported, with the difference factor of 4 or more.
I checked RAMMap and I found that processes never finish. None of the processes that have ever been created finish. They take up memory, which is mostly reported as unused active and page table memory.
For the following I created and exited 100000 cmd.exe processes. I used the following command:
FOR /L %i IN (1,1,100000) DO cmd /c echo %i
Notice the high page table and unused active memory readings.
All exited processes show up in RAMMap
I also noticed that the page table is pretty weird, with sometimes hundreds of physical pages being mapped to the same virtual address, how is that even possible?!
At this point I was able to find more information on this issue, mainly https://randomascii.wordpress.com/2018/02/11/zombie-processes-are-eating-your-memory/
However, diagnosis using FindZombieHandles did not reveal what's holding up these processes.
c:\Programs\blogstuff\FindZombieHandles\prebuilt>FindZombieHandles.exe110416 total zombie processes.6 total zombie threads. 222 zombies held by Taskmgr.exe(6640) 137 zombies of cmd.exe - process handle count: 137 - thread handle count: 0 25 zombies of firefox.exe - process handle count: 25 - thread handle count: 0 11 zombies of dllhost.exe - process handle count: 11 - thread handle count: 0 7 zombies of backgroundTaskHost.exe - process handle count: 7 - thread handle count: 0 5 zombies of RuntimeBroker.exe - process handle count: 5 - thread handle count: 0 5 zombies of EngHost.exe - process handle count: 5 - thread handle count: 0 4 zombies of WmiPrvSE.exe - process handle count: 4 - thread handle count: 0 4 zombies of ctfmon.exe - process handle count: 4 - thread handle count: 0 4 zombies of consent.exe - process handle count: 4 - thread handle count: 0 3 zombies of taskhostw.exe - process handle count: 3 - thread handle count: 0 3 zombies of OpenConsole.exe - process handle count: 3 - thread handle count: 0 2 zombies of mobsync.exe - process handle count: 2 - thread handle count: 0 2 zombies of DbgX.Shell.exe - process handle count: 2 - thread handle count: 0 1 zombie of WindowsTerminal.exe - process handle count: 1 - thread handle count: 0 1 zombie of svchost.exe - process handle count: 1 - thread handle count: 0 1 zombie of StartMenuExperienceHost.exe - process handle count: 1 - thread handle count: 0 1 zombie of sppsvc.exe - process handle count: 1 - thread handle count: 0 1 zombie of SearchHost.exe - process handle count: 1 - thread handle count: 0 1 zombie of pwsh.exe - process handle count: 1 - thread handle count: 0 1 zombie of msedgewebview2.exe - process handle count: 1 - thread handle count: 0 1 zombie of explorer.exe - process handle count: 1 - thread handle count: 0 1 zombie of conhost.exe - process handle count: 1 - thread handle count: 0 1 zombie of audiodg.exe - process handle count: 1 - thread handle count: 0 9 zombies held by svchost.exe(12660) 9 zombies of EngHost.exe - process handle count: 9 - thread handle count: 0 2 zombies held by nvcontainer.exe(4908) 2 zombies of rundll32.exe - process handle count: 2 - thread handle count: 0 2 zombies held by svchost.exe(4620) 2 zombies of explorer.exe - process handle count: 2 - thread handle count: 2 1 zombie held by atieclxx.exe(3504) 1 zombie of atieah32.exe - process handle count: 1 - thread handle count: 1 1 zombie held by NVDisplay.Container.exe(2896) 1 zombie of dbInstaller.exe - process handle count: 1 - thread handle count: 1 1 zombie held by svchost.exe(2268) 1 zombie of userinit.exe - process handle count: 1 - thread handle count: 0ObjExp.exe is also not helpful in this case. It shows the correct number of objects but can't identify the handles (notice how the number of handles is much smaller than the number of objects)
File, Token, and Event objects also follow the count, though I'm not sure how to interpret this, I assume they are just naturally created for every process (for example File handle being the image)
At this point I'm pretty sure the issue does not lie in any user-space process, and instead is caused by some system component/driver.
I attempted searching for these handles with WinDbg hooked up to the kernel with Driver Verifier set to Misc on all drivers. I tried executing !handle 0 3 0 Process but I can't find anything abnormal and the output is so slow (multiple seconds per line) that it's impossible to get a full reading. I also tried !process 0 0 but none of the listed processes show an abnormal amount of handles, though some show 0 handles which I assume may be a permission issue.
At this point I'm at a loss, while this issue is not debilitating it gets pretty annoying for my usual workloads that spawn a lot of processes. After 10 hours of normal computer use yesterday I amounted 20GBs of unused active memory and page table memory (held up by zombie processes), with over 20k processes.
driverquery /v output (and obvious potential offenders? The output is always the same regardless of how many cmd processes I spawn): https://pastebin.com/CcT7rzbJ
edit. The issue is gone in safe mode. This is the diff of the driverquery between normal operation and safemode where the driver is running in normal operation and stopped in safemode:
AmdPPM AMD Processor Driver AMD Processor Driver Kernel Manual Running OK TRUE FALSE 126,976 57,344 0 C:\Windows\system32\drivers\amdppm.sys 12,288 amduw23g amduw23g amduw23g Kernel Manual Running OK TRUE FALSE 78,856,192 7,716,864 0 2024-01-10 22:31:50 C:\Windows\system32\DriverStore\FileRepository\u 12,288 AtiHDAudioSe AMD Function Driver fo AMD Function Driver fo Kernel Manual Running OK TRUE FALSE 126,976 81,920 0 2021-07-30 03:45:07 C:\Windows\system32\drivers\AtihdWT6.sys 4,096 bam Background Activity Mo Background Activity Mo Kernel System Running OK TRUE FALSE 49,152 12,288 0 C:\Windows\system32\drivers\bam.sys 4,096 bfs Brokering File System Brokering File System File System Auto Running OK TRUE FALSE 4,096 36,864 0 C:\Windows\system32\drivers\bfs.sys 4,096 bindflt Windows Bind Filter Dr Windows Bind Filter Dr File System Auto Running OK TRUE FALSE 90,112 24,576 0 C:\Windows\system32\drivers\bindflt.sys 4,096 BTHPORT Bluetooth Port Driver Bluetooth Port Driver Kernel Manual Running OK TRUE FALSE 356,352 1,331,200 0 C:\Windows\system32\drivers\BTHport.sys 8,192 BTHUSB Bluetooth Radio USB Dr Bluetooth Radio USB Dr Kernel Manual Running OK TRUE FALSE 24,576 61,440 0 C:\Windows\system32\drivers\BTHUSB.sys 4,096 cdrom CD-ROM Driver CD-ROM Driver Kernel System Running OK TRUE FALSE 81,920 73,728 0 C:\Windows\system32\drivers\cdrom.sys 4,096 CldFlt Windows Cloud Files Fi Windows Cloud Files Fi File System Auto Running OK TRUE FALSE 339,968 122,880 0 C:\Windows\system32\drivers\cldflt.sys 4,096 FileCrypt FileCrypt FileCrypt File System System Running OK TRUE FALSE 28,672 16,384 0 C:\Windows\system32\drivers\filecrypt.sys 4,096 gameflt gameflt gameflt File System Manual Running OK TRUE FALSE 57,344 28,672 0 C:\Windows\system32\DriverStore\FileRepository\g 4,096 HTTP HTTP Service HTTP Service Kernel Manual Running OK TRUE FALSE 790,528 499,712 0 C:\Windows\system32\drivers\HTTP.sys 12,288 hvsocketcont hvsocketcontrol hvsocketcontrol Kernel System Running OK TRUE FALSE 12,288 8,192 0 C:\Windows\system32\drivers\hvsocketcontrol.sys 4,096 IntcAzAudAdd Service for Realtek HD Service for Realtek HD Kernel Manual Running OK TRUE FALSE 4,079,616 1,339,392 0 2024-01-09 10:10:59 C:\Windows\system32\drivers\RTKVHD64.sys 8,192 kldbgdrv kldbgdrv kldbgdrv Kernel Manual Running OK TRUE FALSE 4,096 4,096 0 2013-04-24 10:35:42 \??\C:\Windows\system32\Drivers\kldbgdrv.sys 4,096 ksthunk Kernel Streaming Thunk Kernel Streaming Thunk Kernel Manual Running OK TRUE FALSE 12,288 16,384 0 C:\Windows\system32\drivers\ksthunk.sys 4,096 lltdio Link-Layer Topology Di Link-Layer Topology Di Kernel Auto Running OK TRUE FALSE 8,192 45,056 0 C:\Windows\system32\drivers\lltdio.sys 8,192 luafv UAC File Virtualizatio UAC File Virtualizatio File System Auto Running OK TRUE FALSE 69,632 12,288 0 C:\Windows\system32\drivers\luafv.sys 12,288 MMCSS Multimedia Class Sched Multimedia Class Sched Kernel Auto Running OK TRUE FALSE 12,288 20,480 0 C:\Windows\system32\drivers\mmcss.sys 8,192 monitor Microsoft Monitor Clas Microsoft Monitor Clas Kernel Manual Running OK TRUE FALSE 40,960 28,672 0 C:\Windows\system32\drivers\monitor.sys 4,096 MsLldp Microsoft Link-Layer D Microsoft Link-Layer D Kernel Auto Running OK TRUE FALSE 16,384 28,672 0 C:\Windows\system32\drivers\mslldp.sys 4,096 MsSecCore Microsoft Security Cor Microsoft Security Cor Kernel Boot Running OK TRUE FALSE 4,096 12,288 0 C:\Windows\system32\drivers\msseccore.sys 4,096 MTKBTFilterx MTK BT Filter Driver MTK BT Filter Driver Kernel Manual Running OK TRUE FALSE 4,096 286,720 0 2022-06-24 13:34:51 C:\Windows\system32\drivers\mtkbtfilterx.sys 4,096 mtkwlex Mediatek PCI LE Extens Mediatek PCI LE Extens Kernel Manual Running OK TRUE FALSE 4,096 1,490,944 0 2023-11-29 22:06:12 C:\Windows\system32\drivers\mtkwl6ex.sys 8,192 Ndu Windows Network Data U Windows Network Data U Kernel Auto Running OK TRUE FALSE 4,096 77,824 0 C:\Windows\system32\drivers\Ndu.sys 4,096 npsvctrig Named pipe service tri Named pipe service tri Kernel System Running OK TRUE FALSE 12,288 12,288 0 C:\Windows\system32\drivers\npsvctrig.sys 4,096 NVHDA Service for NVIDIA Hig Service for NVIDIA Hig Kernel Manual Running OK TRUE FALSE 45,056 40,960 0 2023-01-20 11:48:10 C:\Windows\system32\drivers\nvhda64v.sys 4,096 nvlddmkm nvlddmkm nvlddmkm Kernel Manual Running OK TRUE FALSE 43,884,544 12,451,840 0 2024-04-02 22:30:52 C:\Windows\system32\DriverStore\FileRepository\n 45,056 NvModuleTrac NvModuleTracker NvModuleTracker Kernel Manual Running OK TRUE FALSE 8,192 12,288 0 2022-05-13 15:38:38 C:\Windows\system32\DriverStore\FileRepository\n 4,096 nvvad_WaveEx NVIDIA Virtual Audio D NVIDIA Virtual Audio D Kernel Manual Running OK TRUE FALSE 20,480 8,192 0 2022-09-22 17:10:30 C:\Windows\system32\drivers\nvvad64v.sys 4,096 P9Rdr Plan 9 Redirector Driv Plan 9 Redirector Driv Kernel Manual Running OK TRUE FALSE 4,096 81,920 0 C:\Windows\system32\drivers\p9rdr.sys 4,096 PEAUTH PEAUTH PEAUTH Kernel Auto Running OK TRUE FALSE 577,536 143,360 0 C:\Windows\system32\drivers\peauth.sys 4,096 rspndr Link-Layer Topology Di Link-Layer Topology Di Kernel Auto Running OK TRUE FALSE 8,192 65,536 0 C:\Windows\system32\drivers\rspndr.sys 4,096 Serenum Serenum Filter Driver Serenum Filter Driver Kernel Manual Running OK TRUE FALSE 16,384 12,288 0 C:\Windows\system32\drivers\serenum.sys 4,096 Serial Serial port driver Serial port driver Kernel Manual Running OK TRUE FALSE 40,960 28,672 0 C:\Windows\system32\drivers\serial.sys 4,096 srv Server SMB 1.xxx Drive Server SMB 1.xxx Drive File System Auto Running OK TRUE FALSE 303,104 81,920 0 C:\Windows\system32\DRIVERS\srv.sys 4,096 srv2 Server SMB 2.xxx Drive Server SMB 2.xxx Drive File System Manual Running OK TRUE FALSE 258,048 217,088 0 C:\Windows\system32\DRIVERS\srv2.sys 4,096 srvnet srvnet srvnet File System Manual Running OK TRUE FALSE 94,208 167,936 0 C:\Windows\system32\DRIVERS\srvnet.sys 4,096 storqosflt Storage QoS Filter Dri Storage QoS Filter Dri File System Auto Running OK TRUE FALSE 20,480 40,960 0 C:\Windows\system32\drivers\storqosflt.sys 4,096 storvsp storvsp storvsp Kernel Manual Running OK TRUE FALSE 69,632 61,440 0 C:\Windows\system32\drivers\storvsp.sys 4,096 tcpipreg @%SystemRoot%\System32 @%SystemRoot%\System32 Kernel Auto Running OK TRUE FALSE 4,096 36,864 0 C:\Windows\system32\drivers\tcpipreg.sys 4,096 UCPD UCPD UCPD File System System Running OK TRUE FALSE 4,096 20,480 0 C:\Windows\system32\drivers\UCPD.sys 8,192 Vid Vid Vid Kernel System Running OK TRUE FALSE 401,408 200,704 0 C:\Windows\system32\drivers\Vid.sys 4,096 vmbusr Virtual Machine Bus Pr Virtual Machine Bus Pr Kernel Manual Running OK TRUE FALSE 86,016 126,976 0 C:\Windows\system32\drivers\vmbusr.sys 4,096 vmsmp vmsmp vmsmp Kernel Manual Running OK TRUE FALSE 12,288 1,970,176 0 C:\Windows\system32\drivers\vmswitch.sys 4,096 VMSP VmSwitch Protocol Driv VmSwitch Protocol Driv Kernel Auto Running OK TRUE FALSE 12,288 1,970,176 0 C:\Windows\system32\drivers\vmswitch.sys 4,096 vpcivsp Microsoft Hyper-V PCI Microsoft Hyper-V PCI Kernel Manual Running OK TRUE FALSE 98,304 57,344 0 C:\Windows\system32\drivers\vpcivsp.sys 4,096 vwifibus Virtual Wireless Bus D Virtual Wireless Bus D Kernel Manual Running OK TRUE FALSE 8,192 20,480 0 C:\Windows\system32\drivers\vwifibus.sys 4,096 vwifimp Virtual WiFi Miniport Virtual WiFi Miniport Kernel Manual Running OK TRUE FALSE 4,096 40,960 0 C:\Windows\system32\drivers\vwifimp.sys 4,096 wanarp Remote Access IP ARP D Remote Access IP ARP D Kernel Auto Running OK TRUE FALSE 53,248 24,576 0 C:\Windows\system32\DRIVERS\wanarp.sys 4,096 wcifs Windows Container Isol Windows Container Isol File System Auto Running OK TRUE FALSE 131,072 40,960 0 C:\Windows\system32\drivers\wcifs.sys 8,192 WdFilter Microsoft Defender Ant Microsoft Defender Ant File System Boot Running OK TRUE FALSE 372,736 69,632 0 C:\Windows\system32\drivers\wd\WdFilter.sys 32,768 wdiwifi WDI Driver Framework WDI Driver Framework Kernel Manual Running OK TRUE FALSE 4,096 868,352 0 C:\Windows\system32\DRIVERS\wdiwifi.sys 4,096 WdNisDrv Microsoft Defender Ant Microsoft Defender Ant Kernel Manual Running OK TRUE FALSE 8,192 61,440 0 C:\Windows\system32\drivers\wd\WdNisDrv.sys 4,096 wtd wtd wtd Kernel Auto Running OK TRUE FALSE 28,672 28,672 0 C:\Windows\system32\drivers\wtd.sys 4,096 Xvdd XVDD Port Driver XVDD Port Driver Kernel Manual Running OK TRUE FALSE 4,096 503,808 0 C:\Windows\system32\DriverStore\FileRepository\x 4,096 I'm suspecting bam or wtd, but I have no idea how to diagnose it or even disable them. I also see a possibility that it's caused by the nvidia drivers but that would be very bad.
edit2: poolmon doesn't show anything abnormal. The only thing creating 10000 cmd.exe processes does is put the Proc tagged object into the top 10, which is expected as this handles the Process Objects. I don't think I'll be able to find this leak with poolmon as the handles take negligible amount of space and I'm not even sure if they would be reflected in the mem usage.
edit3: I removed NVIDIA GPU drivers with DDU and reinstalled with NVCleanstall without telemetry. NvModuleTracker is not the issue. It was a possibility since it uses PsSetCreateProcessNotifyRoutineEx, but it turns out it's handling it properly. I don't know how to check which other drivers use this callback...