Help! Someone has deleted events from my Windows event log!

From time to time I hear this, and it usually turns out not to be the case.

I'll begin with a little background.

First, The eventlog service does not have (and never did have) any public or private API to delete individual events- there is a log clear API but nothing else.  The eventlog team thought about implementing selective delete for Vista (there were some internal groups asking for it) but a lot of us security types yelled at them and nothing came of it.  Logs are logs, not databases- if you want selective delete, export the events you want to a database and have at it.

Second, there is no getting around the 10 Immutable Laws of Security, particularly law #6 (a computer is only as secure as the administrator is trustworthy) and law #2 (if a bad guy can alter the operating system on your computer, it's not your computer anymore).

What this means is that, no matter if we implemented the most advanced, coolest, 1337est event-signing/real-time-exporting/writing-to-optical-media-and-a-line-printer-too event system, IT DOESN'T MATTER IF THE ATTACKER IS THE ADMINISTRATOR- all we do is reduce the window of time that that person has to do his dirty work.  Now I will admit there is value in those features, but if such an evil person were to use his powers of debugging to open the services.exe process (where eventlog lives) and inject a thread which alters the eventlog data structures in memory in real time, prior to commit to disk, then none of that stuff would help us.  As a matter of fact such tools exist, they are not theoretical.  I haven't seen the Vista versions yet but there's no technical reason why such a tool could not be built for Vista.

However, the cases I've seen of apparent gaps in event logs have a much more mundane explanation: the "Retain X days" event retention policy.  This is an evil setting; if people truly understood it they wouldn't use it.  Prepare to truly understand it.

<puts on lab coat>

Imagine you have a finite space S to store resources of type R.  You get a constant incoming stream of R's, and put each of them into S.  Now imagine that S is full and a new R arrives.  You have two choices:
 
1.       Throw the new R away.
2.       Remove one or more of the old R's from S (enough so that the new R can fit into S) and put the new R into S.
 
When selecting which old R's to discard, the generally accepted best practice is that you should throw away the oldest R first- in other words freshness is a priority.  Of course if you wanted to optimize for space you could just pick the smallest old R equal to or larger than the new R, but that would cause ordering problems if you wanted to maintain sequential access.  You could even pick one or more old R's at random but that would be too arbitrary for most structured purposes like logging.
 
Now imagine that you had an additional constraint: you can throw away old R's, but only if they're more than X days old.
 
Now your choices are:
 
1.       Throw the new R away, or
2.       Throw away one or more of the old R's, if and only if there are enough R's that are older than X.

If there are no R's older than X, you may not discard any old R's and since you have a fixed size buffer S and there is no room for the new R, you MUST choose option 1- throw away the new R.  You have no other choices.
 
This is the situation with event log.  "Retain X days" actually CAUSES event loss (as does "Overwrite as needed").  However, Overwrite as needed causes predictable event loss (oldest events gone).  Retain X days causes unpredictable event loss (if the log is full and there are no events older than X days, then NEW events are thrown away until there are some events older than X days).

Detecting gaps in your log

Can we detect if someone has deleted events out of your log?

At the end of the day, the event log is an ordered list of data structures (called event records), with each having a pointer to the next.  There is also a unique, monotonically increasing sequence number associated with each event record.

A clever attacker will have disabled the instrumentation that causes the event to be raised; eventlog will never have been involved so there will be no gap (but no event).

A less clever or less resourceful attacker who deletes an event from the log, probably will not go fix up the sequence numbers for the rest of the log (in fact the attacker might not even fix up the pointers, causing the eventlog service to crash or hang).

We can use this priciple to our advantage.  By examining each event and looking at its sequence number, we can look for gaps in the stream and, although we can never be sure that there was no tampering, we can often tell when there was tampering.

Here's a VBScript that demonstrates this concept.  I don't provide VBScript support; you're on your own on this one.

'EventLogGapDetector.vbs

' (c) 2007 Microsoft Corporation, All Rights Reserved

'

strComputer = "."

Set objWMIService = GetObject("winmgmts:\\" & strComputer & "\root\CIMV2")

Set colItems = objWMIService.ExecQuery( _

    "SELECT * FROM Win32_NTLogEvent ",,48)

iPrev = 0

first = true

gapdetected = false

newgapdetected = false

currlogfile = ""

oldlogfile = ""

For Each objItem in colItems

    iCurrent = CInt(objItem.RecordNumber)

    currlogfile = objItem.Logfile

    if ((iCurrent <> (iPrev-1)) and (not (first)) and currlogfile=oldlogfile) then newgapdetected = true

    if (newgapdetected) then Wscript.Echo "Gap detected, log file = " & currlogfile & ", last record = " & iPrev & ", current record = " & objItem.RecordNumber

    if (newgapdetected) then gapdetected = true

   iPrev = CInt(objItem.RecordNumber)

    first = false

    newgapdetected = false

    oldlogfile = currlogfile

Next

if not (gapdetected) then Wscript.Echo "No gaps detected."

Eric

The information provided in this post is provided "AS-IS" with no warranty, and confers no rights.