G
GVIceDevMan
We have a customer who is running our application on a suite of Dell 730 servers. The application is responsible for capturing, processing and broadcasting SD & HD resolution IP video streams using a custom network bypass driver (that we wrote) that runs as Windows service/kernel driver. Our customer has been noticing periods when the application fails to process and broadcast video in a timely manner, leading to black video and silent audio at the receiving equipment.
We have been remotely connecting to one of the machines to monitor its performance and to see if we can identify when the issues occur and what might be causing them. We have been using Remote Desktop (RDP) to connect in via a VPN tunnel. The system appears to be running with a total CPU usage of around 33% for most of the time and, once logged in, the system has functioned without any issues all day for a number of days in a row. However there are other periods when the total CPU usage is 100% and these periods correlate with periods when we see issues with the output from the application. For most of the time the servers run with no one logged in with the application preforming the same series of operations repeatedly with not obvious fluctuation in application load.
All unnecessary software and Windows services have been removed or disabled, leaving just the application executable (there is one for media validation and one for video/audio processing), the network bypass driver and essential Windows services. Performance Monitor logs show that neither ICE executable is the one responsible for consuming all the CPU time during the periods when the CPU reports 100% utilisation. In fact the Perf Mon logs show no application using all the CPU time, with the bulk of the total CPU time reported as ‘System Idle’ time.
We are struggling to explain what is causing this problem. However, we have noticed that we can trigger the 100% CPU utilisation state by logging into (or out of) the ICE machine via RDP. On one occasion the system ‘latched’ into the 100% state and (according to the PerfMon logs) remained in this state from 5PM to 9AM the following day, only returning to a 33% CPU total when someone logged back in via RDP.
We have been unable to obtain a Dell 730 server so we have a Dell 740 server in our lab running the same application software and similar system build but, so far, we have been unable to reproduce this problem. We have contacted Dell and, at their suggestion, are updating the server BIOS and drivers.
Can you suggest what might be causing the system to enter this 100% CPU utilisation state?
Are there parameters we can set that might prevent RDP from triggering this problem?
Continue reading...
We have been remotely connecting to one of the machines to monitor its performance and to see if we can identify when the issues occur and what might be causing them. We have been using Remote Desktop (RDP) to connect in via a VPN tunnel. The system appears to be running with a total CPU usage of around 33% for most of the time and, once logged in, the system has functioned without any issues all day for a number of days in a row. However there are other periods when the total CPU usage is 100% and these periods correlate with periods when we see issues with the output from the application. For most of the time the servers run with no one logged in with the application preforming the same series of operations repeatedly with not obvious fluctuation in application load.
All unnecessary software and Windows services have been removed or disabled, leaving just the application executable (there is one for media validation and one for video/audio processing), the network bypass driver and essential Windows services. Performance Monitor logs show that neither ICE executable is the one responsible for consuming all the CPU time during the periods when the CPU reports 100% utilisation. In fact the Perf Mon logs show no application using all the CPU time, with the bulk of the total CPU time reported as ‘System Idle’ time.
We are struggling to explain what is causing this problem. However, we have noticed that we can trigger the 100% CPU utilisation state by logging into (or out of) the ICE machine via RDP. On one occasion the system ‘latched’ into the 100% state and (according to the PerfMon logs) remained in this state from 5PM to 9AM the following day, only returning to a 33% CPU total when someone logged back in via RDP.
We have been unable to obtain a Dell 730 server so we have a Dell 740 server in our lab running the same application software and similar system build but, so far, we have been unable to reproduce this problem. We have contacted Dell and, at their suggestion, are updating the server BIOS and drivers.
Can you suggest what might be causing the system to enter this 100% CPU utilisation state?
Are there parameters we can set that might prevent RDP from triggering this problem?
Continue reading...