J
Jim
On Dec 31, I posted about a problem I was having with terminal services
sessions failing to log on. I wrote:
"We recently installed SP2 on a base 2003 standard server via windows
update.
Since the installation, we receive "warning" event 244 ("Failed to create a
desktop due to desktop heap exhaustion") and all subsequent logons of remote
desktops fail, with no further error messages. One or possibly two logons
work, then the the warning message is issued, and further logons fail until
the system is rebooted...."
I received a number of useful suggestions, which I appreciate including
detailed instructions on how to install dheapmon (that worked, BTW).
I took about a week but I finally found the problem.
I thought I'd share the solution for future reference, now that I'm certain
that I've resolved the problem (no failures for almost 3 weeks).
The problem was not due to simple "heap exhaustion," or a too small memory
allocation size, but to a trojan horse and/or virus that Symantec Anti-virus
didn't completely remove.
There are several variations of this virus (Infostealer.Banker) and a
similar one (Trojan.Gpcoder) both of which were installed on this server.
Both inject malicious code into WINLOGON.EXE which prevents the terminal
sessions from being created as well as create bogus folders to contain the
stolen data they obtain.
SAV's logs noted the detection and claimed the viruses were cleaned. Not
true, it appears. And I'm a bit disappointed that SAV didn't (a) remove
them completely, and (b) didn't tell me it didn't remove them, but told me,
instead, that it did and that they were only found in the IE cache.
Neither virus was totally activated, though winlogon was corrupted within
hours of rebooting the system.
Repeated manual A/V scans did not show the virus from the System Center
view. However, a manual live-update followed by a manual scan from the
server using the client software, triggered a detection followed by a clean
up of 17 files plus a reboot. Subsequent manual scans show the machine to be
clean.
How did this happen? One of two possiblilities, I think.
One: The person (not me) who actually installed 2003 sp2 also installed IE
7. IE 7's "demo" and intro script for new users *DISABLES* the usual
automatic block of websites not in the trusted list (which, BTW, only
includes Microsoft/Symantec/ and the server vendor's support sites on this
server).
If you cancel the demo/intro, say because you neither need or want to see
all about the new features of IE 7, *THE BLOCK* is not in place and anybody
can access any website, even those not in the trusted list, until you watch
the G*D* demo!
Visit a corrupted site (and there are legions of them these days), and you
can get your computer infected. So, if someone used the web browser on the
server to browse the web (and nobody will own up to doing this, BTW), you
can infect your server.
Bottom line, if you MUST install IE7, run the demo before you walk away.
Two: Using your server to read your email. Outlook Express 7 comes with IE7.
Download infected email with "preview" on and wallah! Especially if the
email is HTML formated.
Symantec A/V Business pack was current at the time and up to date with
respect to signatures. Downloading and pushing sig files and scanning
servers and desktops under its management is supposed to be automatic and
the signature files did catch the virus in the IE cache, multiple times, it
appears. Other than winlogon (which did show up as infected during the final
manual scan), none of the other symptoms (the installed folders in
%system/system32, the executable programs (NTSO.exe in memory) or registry
entries were present. By viewing the various logs I maintain, I have
reasonable confidence that the "mission" of the trojans was not accomplished
and that the server was not actually compromised.
Since I manually completed the removal of the trojans, I have had no similar
failures of establishing sessions. Repeated, daily, manual scans shows no
new threats.
Jim
sessions failing to log on. I wrote:
"We recently installed SP2 on a base 2003 standard server via windows
update.
Since the installation, we receive "warning" event 244 ("Failed to create a
desktop due to desktop heap exhaustion") and all subsequent logons of remote
desktops fail, with no further error messages. One or possibly two logons
work, then the the warning message is issued, and further logons fail until
the system is rebooted...."
I received a number of useful suggestions, which I appreciate including
detailed instructions on how to install dheapmon (that worked, BTW).
I took about a week but I finally found the problem.
I thought I'd share the solution for future reference, now that I'm certain
that I've resolved the problem (no failures for almost 3 weeks).
The problem was not due to simple "heap exhaustion," or a too small memory
allocation size, but to a trojan horse and/or virus that Symantec Anti-virus
didn't completely remove.
There are several variations of this virus (Infostealer.Banker) and a
similar one (Trojan.Gpcoder) both of which were installed on this server.
Both inject malicious code into WINLOGON.EXE which prevents the terminal
sessions from being created as well as create bogus folders to contain the
stolen data they obtain.
SAV's logs noted the detection and claimed the viruses were cleaned. Not
true, it appears. And I'm a bit disappointed that SAV didn't (a) remove
them completely, and (b) didn't tell me it didn't remove them, but told me,
instead, that it did and that they were only found in the IE cache.
Neither virus was totally activated, though winlogon was corrupted within
hours of rebooting the system.
Repeated manual A/V scans did not show the virus from the System Center
view. However, a manual live-update followed by a manual scan from the
server using the client software, triggered a detection followed by a clean
up of 17 files plus a reboot. Subsequent manual scans show the machine to be
clean.
How did this happen? One of two possiblilities, I think.
One: The person (not me) who actually installed 2003 sp2 also installed IE
7. IE 7's "demo" and intro script for new users *DISABLES* the usual
automatic block of websites not in the trusted list (which, BTW, only
includes Microsoft/Symantec/ and the server vendor's support sites on this
server).
If you cancel the demo/intro, say because you neither need or want to see
all about the new features of IE 7, *THE BLOCK* is not in place and anybody
can access any website, even those not in the trusted list, until you watch
the G*D* demo!
Visit a corrupted site (and there are legions of them these days), and you
can get your computer infected. So, if someone used the web browser on the
server to browse the web (and nobody will own up to doing this, BTW), you
can infect your server.
Bottom line, if you MUST install IE7, run the demo before you walk away.
Two: Using your server to read your email. Outlook Express 7 comes with IE7.
Download infected email with "preview" on and wallah! Especially if the
email is HTML formated.
Symantec A/V Business pack was current at the time and up to date with
respect to signatures. Downloading and pushing sig files and scanning
servers and desktops under its management is supposed to be automatic and
the signature files did catch the virus in the IE cache, multiple times, it
appears. Other than winlogon (which did show up as infected during the final
manual scan), none of the other symptoms (the installed folders in
%system/system32, the executable programs (NTSO.exe in memory) or registry
entries were present. By viewing the various logs I maintain, I have
reasonable confidence that the "mission" of the trojans was not accomplished
and that the server was not actually compromised.
Since I manually completed the removal of the trojans, I have had no similar
failures of establishing sessions. Repeated, daily, manual scans shows no
new threats.
Jim
