David Wang wrote an excellent post about this in 2006: http://blogs.msdn.com/david.wang/archive/2006/02/09/Can-I-write-an-ISAPI-Filter-using-Managed-Code.aspx
This includes writing ISAPI code that even USES managed components. Anything that needs to load the CLR will result in poor performance of the ISAPI and worse, result in ASP.NET not functioning correctly.
ASP.NET loads the CLR when it initializes. During this loading it does many things to configure the CLR. One example is that it creates on GC thread for each processor on the system. It also sets up some defaults for the thread pools. When you write your ISAPI in managed code or include a COM Visible .NET component through COM in your code, your ISAPI loads the CLR. Since only one CLR can be loaded per process, ASP.NET will not create and configure the CLR and you are left with a severely crippled system.
This problem can also occur if you write an ISAPI Extension. Your Extension could load the CLR runtime if you hit a static file first, initialize the CLR in the ISAPI Extension by using .NET or a COM Visible .NET object and then hit your ASP.NET application after this initialization.
You can clearly see this in a multi processor machine if you create a filter that uses a Managed code COM component. Without your filter in place, hit an aspx page on the site and use DebugDiag to take a dump of the w3wp.exe process. Then analyze this dump with DebugDiag. You will see a GC thread initialized for each processor core installed on your server. On my server I have a quad proc CPU and I see 4 threads with this in the call stack: mscorwks!SVR::gc_heap::gc_thread_function.
Then I created a simple ISAPI filter from one of the existing Platform SDK samples and simply instantiated a COM Visible .NET component. I installed the ISAPI filter and re-ran my test. As expected, I saw only 1 thread with mscorwks!SVR::gc_heap::gc_thread_function.
This problem manifests itself not only with the GC threads but all the information set by the processModel (see http://msdn.microsoft.com/en-us/library/7w2sway1.aspx). In theory you could come up with some sort of scheme to read the machine.config file and parse and set this information, but this is untested and definitely not supported (read: you are on your own).
If you really are in need of writing Managed Code, you should investigate the Managed Pipeline in IIS 7 and use that http://msdn.microsoft.com/en-us/library/ms227673.aspx. There are a couple of other workarounds you could explore like out of proc components etc… but these would all create performance issues for a Filter or Extension.
Please let me know if this blog post was useful to you!