Raymond today has a discussion up about the folly of trying to set security with a granularity of per-DLL. As he explains, the moment you let something untrusted run in your process space, you cannot trust anything in the process. You cannot wall off a DLL or a section of code from the rest of the process. "That's why code is not a security principal," he concludes.
But you can wall off a process. There is an ACL associated with each running process (via its user context), so you can restrict what code can do depending on its process. More on that in a moment.
The other point Raymond drives home is that, to the system, there is no difference between the application and the user. Put another way, When an application has the credentials of a user and attempts to perform an operation, there is no good way for the system to know whether the app asked the user before performing it. Therefore, the only effective way to limit what an application can do is reduce its process ACL when the application is launched, and then limit what it can do based on that.
So consider this in the context of the Dancing Bunnies problem. There is a website which will let me download an executable to watch dancing bunnies. Conventional security wisdom says that the only way to be secure is to not download the executable. This is obviously the most secure choice, from a technical standpoint, but the technical standpoint goes on its ear when you add a user to the equation. Conventional security wisdom doesn't let me see my dancing bunnies, so I'm going to download it anyway. When I run the executable, it takes on my user token, and suddenly the website's code has the full ability to do anything to my system that I can do -- and if I'm on a pre-Vista OS, that means admin access. What's worse, it probably won't even bother to ask whether I want my system owned. The nerve!
So somehow we lost the most secure option (not downloading the exe) because it wasn't acceptible to the user -- I couldn't see my dancing bunnies. Without that most secure option, the only one left is the least secure (system owned). What I think we need is another status for the app. I want an execution option that says "I want to see my dancing bunnies, but I don't want my system owned". UAC on Vista does a very good job of preventing the system owned result, but a malicious program can still do a lot of damage. Even better is the "low" security context that IE7 runs in by default on Vista, which can only access certain directories and APIs. This mode is effectively the application-level security that I spoke of earlier.
What I'd like to see, as a partial answer to the Dancing Bunnies problem, is the ability to download and save an app from the internet, and have that app marked as "untrusted". When I launch the app, it launches in some sort of sandbox similar to the "low" security mode with restricted process permissions. If it's coded correctly then it shows me my dancing bunnies. If it's malicious, then I haven't lost anything. This is something that users will buy into.
Of course, we still have the social engineering problem of exes that tell the user they can only see dancing bunnies after they click away that nagging UAC prompt. The way to fix that is to convince them that Microsoft is more trustworthy than Ye Olde Maliciouse Website. When I come up with the solution to that one, I'll let you know.