How do I cancel an IRP that another thread may be completing at the same time?

Let's say that you allocated a PIRP and sent it down your device stack.  You free the PIRP in the completion routine and then return STATUS_MORE_PROCESSING_REQUIRED.  To make life more fun, you decide that you want to be able to cancel the sent IRP after you have sent it so you try to do it simple like this

 typedef struct _DEVICE_EXTENSION {
    KSPIN_LOCK SentIrpLock;
    PIRP SentIrp;
} DEVICE_EXTENSION;

Sending thread:

 KeAcquireSpinLock(&devext->SentIrpLock, ...);
devext->SentIrp = Irp;
KeReleaseSpinLock(&devext->SentIrpLock, ...);

Canceling thread:

 KeAcquireSpinLock(&devext->SentIrpLock, ...);
if (devext->AllocatedIrp != NULL) {
   IoCancelIrp(devext->SentIrp);
}
KeReleaseSpinLock(&devext->SentIrpLock, ...);

Completion routine:

 PIRP irp;

KeAcquireSpinLock(&devext->SentIrpLock, ...);
irp = devext->SentIrp;

 devext->SentIrp = NULL;
KeReleaseSpinLock(&devext->SentIrpLock, ...);

IoFreeIrp(irp);

return STATUS_MORE_PROCESSING_REQUIRED;

And it then deadlocks ;). If the call to IoCancelIrp causes the IRP to be completed in the calling context (e.g. the one which has acquired the lock), the completion routine will run and try to acquire the lock (SentIrpLock) on the same thread which holds it.

So, life is not that simple and you have to do something more. The basic solution is that you need extra state to track who is touching the PIRP and who can free it. Walter Oney's book has a solution (IIRC, it is in the self initiated I/O section, but I do not have the book handy), but IMHO it is a bit complicated. KMDF has a solution to this problem which I like much more (imagine that ;)).

You need an extra LONG, calling it CompletionCount per PIRP that you want to be able to cancel.

1. You initialize CompletionCount to 1 before sending it down the stack and storing it in devext.
2. Whenever there is a thread that wants to cancel the PIRP, it tries to interlocked increment CompletionCount only if and only if the current CompletionCount value is > 0. For this you need to roll your own InterlockedIncrementWithFloor which is fortunately not that hard and I have already shown you how to do that, https://blogs.msdn.com/doronh/archive/2006/12/06/creating-your-own-interlockedxxx-operation.aspx.
3. After the canceling thread has called InterlockedIncrementWithFloor and IoCancelIrp, it calls InterlockedDecrement.
4. Whomever wants to complete the PIRP, like the completion routine, interlock decrements CompletionCount.

If the returned value from InterlockedDecrement is zero, the caller can complete the PIRP. If not, somebody else is trying to touch the PIRP and you must leave the PIRP alone.  So here is the revised code:

 typedef struct _DEVICE_EXTENSION {
    KSPIN_LOCK SentIrpLock;
    PIRP SentIrp;
    ULONG CompletionCount;
} DEVICE_EXTENSION;

Sending thread:

 KeAcquireSpinLock(&devext->SentIrpLock, ...);
devext->CompletionCount = 1;
devext->SentIrp = Irp;

 KeReleaseSpinLock(&devext->SentIrpLock, ...);

Canceling thread:

 PIRP irp = NULL;
KeAcquireSpinLock(&devext->SentIrpLock, ...);
if (devext->AllocatedIrp != NULL && MyInterlockedIcrementeWithFloor(&devext->CompletionCount, 0) > 0) {
   irp = devext->SentIrp;
}
KeReleaseSpinLock(&devext->SentIrpLock, ...);

if (irp != NULL) {
    IoCancelIrp(irp);
    if (InterlockedDecrement(&devext->CompletionCount) == 0) {
        IoFreeIrp(irp);
    }
}

Completion routine:

 PIRP irp = NULL;

KeAcquireSpinLock(&devext->SentIrpLock, ...);
irp = devext->SentIrp;devext->SentIrp = NULL;
KeReleaseSpinLock(&devext->SentIrpLock, ...);

if (InterlockedDecrement(&devext->CompletionCount) == 0) {
    IoFreeIrp(irp);
}

return STATUS_MORE_PROCESSING_REQUIRED;

The beauty of this solution is that if you add more actors (let's say a timer for an async timeout, all you have to do is bump the CompletionCount to account for them to asynchronously rundown if you cannot cancel them.