High memory usage on Cluster service RHS.exe process on a 4 node Cluster the rhs.exe process claims 240GB memory per node

[Edwin van Halm]

The Cluster was setup 2 months ago, we started to migrate vm to the cluster, every node has 384GB of memory. After i passed the 100 vm's i noticed a total of 97% memory usage in the hyper-v Cluster. The Cluster is reported as healthy, but i have a serious problem because i can not drain any vm's anymore. Here is some information on the cluster, i hope someone can help me and explain why this RHS.exe proces is exhausting the Cluster memory.

- WIndows 2016 - Windows Malicious Software Removal Tool x64 - March 2018 (KB890830) - 2018-03 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4088787) -2018-02 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4074590)

- 103 VM using ,646 GB allocated to VM's

Some messages from the cluster.log

Line 592878: 00002ee4.00006428::2018/05/08-16:26:35.925 ERR [RHS] Error 2147943568 from ResourceControl 17834261 for resource Cluster Virtual Disk (Zilver-volume-8).
Line 592882: 00002ee4.0000437c::2018/05/08-16:26:35.928 ERR [RHS] Error 2147943568 from ResourceControl 17834261 for resource Cluster Virtual Disk (Zilver-volume-4).
Line 592886: 00002ee4.00006428::2018/05/08-16:26:35.931 ERR [RHS] Error 2147943568 from ResourceControl 17834261 for resource Cluster Virtual Disk (Zilver-volume-6).
Line 592890: 00002ee4.0000437c::2018/05/08-16:26:35.933 ERR [RHS] Error 2147943568 from ResourceControl 17834261 for resource Cluster Virtual Disk (Zilver-volume-7).
Line 592894: 00002ee4.00006428::2018/05/08-16:26:35.936 ERR [RHS] Error 2147943568 from ResourceControl 17834261 for resource Cluster Virtual Disk (Gold-volume-2).

I have created a cluster diagnostic info which i can send it by wetranfer

Continue reading...