Sometime ago a customer of mine faced the following issue on his .NET Framework 3.5 app: System.DirectoryServices.AccountManagement.UserPrincipal.IsMemberOf returned false negatives when the group had more than 1500 members. This means that the function returned false when the user was actually in the group. But everything worked properly with VBScript and ADSI.
This turned out to be a limitation on .NET 3.5, which doesn't implement range retrieval for big AD groups (more than 1000/1500 members depending on server version).
My customer ended up applying this really simple workaround: get all groups for a user (around 100 or so in his case), and look for the desired group in that array of groups.
Note that this is not an issue on .NET Framework 4.0, which implements range retrieval now.
I hope this helps.
Alex (Alejandro Campos Magencio)