Dying Thread on Trackbacks, Referrals and Pingbacks


Part 2 of 2


Bug 2: TrackingHandler Thread Dies


Another problem that Scott Hanselman informed me of was that he would frequently stop receiving Trackbacks, Pingbacks and Referrals on his posts. Furthermore, it was intermittent. This was troubling since losing a Trackback means it’s lost forever. Well we went hunting in the code, and thanks to some UnitTest of a theory I had found the answer.


Basically the situation is this. Scott gets a lot of traffic. More than I do. There is a thread in dasBlog that sits around waiting for Trackbacks and the like. You use it by calling trackingQueue.Enqueue(tracking) and then trackingQueueEvent.Set(). So basically dasBlog can sit there and queue a bunch of trackings, and when it’s ready the thread runs to execute them. The code looks like this:


private void TrackingHandler( )
{
    while ( true )
    {
        Tracking tracking;

        trackingQueueEvent.WaitOne();
        while ( true )
        {
            lock( trackingQueue.SyncRoot )
            {
                tracking = trackingQueue.Dequeue() as Tracking;
            }
            if ( tracking != null )
            {
                try
                {
                    InternalAddTracking( tracking );
                }
                catch (Exception e)
                {
                    ErrorTrace.Trace(TraceLevel.Error,e);
                }
            }

            if ( trackingQueue.Count == 0 )
            {
                break;
            }
        }
    }
}


The objects below are created like so:


trackingQueue = new Queue();
trackingQueueEvent = new AutoResetEvent(false);
trackingHandlerThread = new Thread(new ThreadStart(this.TrackingHandler));
trackingHandlerThread.IsBackground = true;
trackingHandlerThread.Start();


So, can you figure out what is wrong? Well I created a unit test that called this 100 times. What I quickly found out was that even though the code was calling break when the trackingQueue.Count was equal to zero the trackingQueueEvent.WaitOne() call wasn’t blocking the while loop from continuing. This caused trackingQueue.Dequeue() to throw an unhanded exception (which should have been in a try catch anyway).


Not knowing a whole lot about this kind of threading I looked at a couple of docs and found the answer. Before calling break I added trackingQueueEvent.Reset(). Problem fixed (I hope).

Comments (3)

  1. Dean Harding says:

    You shouldn’t be using an event in this situation, you should be using a counting semaphore. This is because there’s a race condition between where you check if the queue is empty and resetting the event – someone could add something to queue, set the event, and you’ll just reset it. It’s no big deal, since you’ll pick up the tracking event the next time someone comes along, but it’s still not optimal.

    The reason it wasn’t working before is because of another race condition in the code you posted. If a thread queues a trackback, it’ll set the event and the tracking thread will wake up. Now lets assume that during your call to the InternalAddTracking method, someone else comes along and enqueues a trackback. They’ll enqueue the trackback and set the event as well. But you then loop back and dequeue the next trackback (that they just added) without resetting the event.

    By resetting the event before the break, you’ll fix the crash (since it won’t try to dequeue something that’s not there) but if, between your call to trackingQueue.Count and actually resetting the event, someone else comes along and enqueues a new trackback (and sets the event), then you’ll be resetting the event when there’s a waiting trackback. Like I said, this isn’t a big deal, since you’ll just pick it up when the /next/ person adds a trackback, but maybe that won’t be for some time, and you don’t want the trackback sitting around for too long.

    Another possible solution might be to reset the event as soon as you dequeue an item (i.e. after the call to Dequeue). I think this is OK as well, but I’m too tired to be 100% sure right now 😉

    The usual solution for this problem is like I said above – a semaphore. Basically, a semaphore is a mutex which has a count – when the count is greater than 0, the semaphore is "unsignalled", when = 0, the semaphore is "signalled" (actually, it’s the other way around – a /mutex/ is a /semaphore/ with a maximum count of 1.)

    Anyway, for some reason .NET doesn’t include a native implementation of semaphores, but you can emulate it using the Monitor class. See: http://weblogs.asp.net/tspascoal/archive/2004/01/16/59340.aspx for example.

  2. Adam Weigert says:

    Hey, couldn’t you use a custom managed thread pool that uses a counting semaphore to acquire a multi-threaded queue processor?

    I know Scott was playing with something like that before the whole Intelligent thing came along and he was using SourceGear’s VaultPub.

    I’ve had good luck with that model so far.

  3. Dying Thread on Trackbacks, Referrals and Pingbacks