If I had to choose one performance optimization technique for a web site that uses SSL, such that it gives me the biggest value compared to a relatively small effort (the proverbial low hanging fruit), the answer would be simple – improve cacheability of your content by separating secured and unsecured content into different folders.
From a performance perspective, SSL handshake and encryption/decryption create an additional processing layer, beyond regular HTTP processing (for those who are interested, there is a good article at http://www.microsoft.com/technet/prodtechnol/isa/2004/plan/ssl_performance.mspx for SSL capacity planning). So it only makes sense that you should only use Secure Sockets Layer (SSL) for pages that require it. By having image files and other content that does not have to be secured in a separate folder, you not only avoid the SSL overhead such as encryption/decryption, but also can leverage client and/or server side caching of that content.
For other recommendations on how to improve performance of a web site, check out “Checklist: ASP.NET Performance” at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnpag/html/scalenetcheck03.asp.