Pawel recently looked into a customer report about XslCompiledTransform failing with a StackOverflowException after an upgrade to .NET Framework 3.5 SP1. Same stylesheet, same everything, but now the exception was showing up when it wasn't before.
There were a number of interesting gotchas associated with this.
- The default stack size is smaller when running under IIS, which can lead to spending time figuring out whether you actually have the correct repro or not if you're running in a standalone app.
- It turns out that the stylesheet had always been close to the stack limit, but a change in the JIT to improve speed increased stack usage some, and pushed the stack over the limit.
- There are ways to rewrite the stylesheet to produce the same results but not consume as much stack internally (yay! - no configuration changes needed)
For the real-deal details, I'll refer you to the MSDN Forum thread again.