Just got bitten by this. Spent the better portion of an hour trying to figure out why I was getting cryptic "That assembly does not allow partially trusted callers" errors in my SharePoint logs. Was following some MSDN documentation about this, and placed my assembly in the bin directory of SharePoint. Once I moved my assembly to the GAC, the errors went away (assemblies in the GAC cannot be loaded with partial trust, if my memory serves me right).
The error seemed to stem from me trying to access the ExcelWebRenderer.WokbookUri property - the code would just not run and SP would give me one of the wonderful generic error pages. Other properties on the same control worked okay.
I swear, sometimes debugging SharePoint stuff is like trying to use a rolled piece of paper to hammer a 4 inch steel nail into a concrete wall while blindfolded, with both arms tied behind your back and with the actual nail & said piece of rolled paper located in Texas while you are snugly situated inside a collapsed ice-cave in the Himalayas with only 6 hours of air left, while you are getting overly stressed because you can hear your best friend gulping copious amounts of air.
The drawback of putting it in the GAC, of course, is that it does not get shadowed and thus you cannot overwrite the DLL w/o restarting the SharePoint app-domain. Better the devil I know...