Finally! Something interesting!

OK here’s a fun one! I just got back from lunch to discover an outage on one of our exchange servers.

To be more specific an outage on a particular mail store. All other stores on that server were fine. The boys had kicked the server over but that didn’t resolve anything. All five databases in the store were dismounted when the server came back online.

Finally! Something interesting!

We got the stores back online by forcing them to ignore missing log files (we could always bring back lost mail from the archiving server, yes we do send every piece of email we receive to an archive server doesn’t everyone?). Then we began collating data on the outage and pretty quickly a picture started forming.

I’ll list these in reverse chronological order so as to build the suspense!

Event ID 104 – 4/19/2010 12:26
MSExchangeIS (3240) Mail P-T: The database engine stopped the instance (3) with error (-1090).

For more information, click ....

Event ID 486 – 4/19/2010 12:26
MSExchangeIS (3240) Mail P-T: An attempt to move the file "C:\EXCHANGE LOGS\Mail P-T\E02.log" to "C:\EXCHANGE LOGS\Mail P-T\E020005DC2D.log" failed with system error 2 (0x00000002): "The system cannot find the file specified. ". The move file operation will fail with error -1811 (0xfffff8ed).

For more information, click ....


Event ID 259 – 4/19/2010 12:21
The file C:\EXCHANGE LOGS\Mail P-T\E02.log contains the Malformed Archive Trojan. No cleaner available, file deleted successfully. Detected using Scan engine version 5400.1158 DAT version 5955.0000.

This looks to me like our Anti Virus product found what it thought was an infected file E02.LOG and deleted it; as a result exchange lost the plot! As would I if someone removed one of my transaction logs midflight.

So yeah new policy, exclude your exchange transaction logs from your Anti Virus scans!