A single-instance program is its own denial of service


There are many ways a program can establish itself as a single-instance program; I won't go into them here. But once people head down the path of a single-instance program, they bump into another issue in their security review: Denial of service.

We are using a named mutex with a fixed name in order to detect whether another copy of the program is running. But that also means an attacker can create the mutex first, thereby preventing our program from running at all! How can I prevent this type of denial of service attack?

If the attacker is running in the same security context as your program is (or would be) running in, then there is nothing you can do. Whatever "secret handshake" you come up with to determine whether another copy of your program is running, the attacker can mimic it. Since it is running in the correct security context, it can do anything that the "real" program can do.

But look at the big picture: A single-instance program is its own denial of service! After all, the first instance of the program is preventing the second instance from running. Your program requirements are themselves a security vulnerability. Consequently, you cannot protect against yourself perfectly against a denial of service since a denial of service is what you want in the first place.

What you can do is to understand and narrow the scope of your vulnerability. Clearly you can't protect yourself from an attacker running at the same security privilege, but you can still protect yourself against unprivileged attackers running at other security privileges. This means using securable objects as part of your handshake. Non-administrative users should not be able to prevent other users from running the program, for example. The worst thing that non-administrative users should be allowed to do is to make their own lives miserable.

Comments (35)
  1. BryanK says:

    And for people wondering how to set up an ACL on the named mutex, see this sample code page on MSDN (at least, I assume it works; I’ve never actually used it yet):

    http://msdn.microsoft.com/library/default.asp?url=/library/en-us/secbp/security/creating_a_dacl.asp

    That function takes a pointer to the SECURITY_ATTRIBUTES structure that you later pass to CreateMutex (or OpenMutex, but the Create version is probably preferred for single instance detection).

  2. Interesting.  I’ve written single-instance programs in the past, and the thought never entered my mind that someone might intentionally use my mutex handle to prevent my program from running.  I guess my mind isn’t malicious enough to think of things like this.  I’ll have to keep this in mind the next time I consider a single-instance program.

  3. Mike says:

    This is an interesting dilemma. I have myself a program that needs to be the single instance per winsta. It registers a few kbdhooks, and performs a system-wide well defined and most certainly useful function (ok, it’s a volume control). This way it works for all desktops I create too. I use a global atom as the named object.

    As I obviously also cannot safeguard against the mentioned scenario (nothing can, not even things like e.g. Task Manager, that I suspect must use a desktop-local object, as even that you can not bring up another instance of it on  another desktop using the kbd shortcut, you can get another instance starting if you run taskmgr.exe from e.g. a command prompt on another desktop – or is this a sign of things started by winlogon having named object created in another "instance of something" and therefore is not checked when checking for such a named object from another desktop?), is a global atom the best option for this scenario? I mean, let’s say two or more clients are connected to a terminal server^2-running machine, would the installed hooks and the named object I check for be confined to the same scope (namespace lookup and hooking behaviour)? Would a NULL/non-null security context make a difference, should I switch to (mis-)using a mutex instead?

  4. Gabe says:

    I believe most programs single-instance themselves by just looking for their window class. If they see a window with that class already in existence, they bring it to the foreground and exit, otherwise they open their own window. This makes the instance per-desktop.

    Mac OS (prior to version X, at least) had a much different way of single-instancing all programs. It would look at the file, and if that file was already running, it would just switch to the running version instead of launching it. This meant that you could run multiple instances of an app just by copying the main executable.

    That brings up an interesting point. Instead of naming your mutex/atom/window class/whatever a constant that any attacker can compile into their program, you can use your EXE’s path. That means any attacker would have to somehow determine at runtime what your shared "secret" is, which would be almost impossible when you consider your program could run from the network. This method also allows the user to override your single-instancing by creating a copy of the EXE and running that.

  5. Teis Johansen says:

    One particular problem I have come across with the global mutex approach is when running as a low-privilege user and an administrator, or more generally when there are multiple users logged in. If you use a global mutex in to ensure a single instance, your application will seem to fail (ie. never launch) for the second user to launch it. I have come across this behavior even in expensive commercial applications (yes, I run as a low-privilege user and yes, even when writing software)

  6. Sean W. says:

    I’ve thought for a while now that the whole notion of a single-instance program was an inherently bad one, and even though I’ve installed thousands of programs over the years, I’ve yet to find one where single-instance behavior would be better than multiple-instance behavior from a usability perspective.  Denial of service indeed:  Single-instance software does little more than deny the user his right to use his software how he wants.  I think the easiest way around this "security hole" is to just design your software to allow multiple instances.

    (Oh, and Raymond, you have a minor thinko in one of your sentences:  "Consequently, you cannot protect AGAINST yourself perfectly against a denial of service…")

  7. Boris Zakharin says:

    I disagree. What if you are an installer? Or an audio player or a CD burning software? What if you are a Windows shell or the display properties control panel?

  8. I was wondering. You can create DLL’s with shared memory using following statement in the DEF file:

    SECTIONS .SECTIONNAME READ WRITE SHARED

    If you use that block of memory to e.g. keep a reference count; wouldn’t that be a fail safe way of implementing this feature? This cannot be hijacked if you, e.g., check the path of the modules that increase or decrease the ref count.

    Am I missing something?

    [You’re replacing one security hole with an even bigger security hole. -Raymond]
  9. Sean W. says:

    Boris:  Explain to me why any of those should be prohibited from having multiple instances.  I can come up with legitimate reasons for each one you named to have multiple instances:

    Installer: You want multiple installations to different directories for different users, and to save time, you want to run the installations in parallel (this kind of time saving is especially significant in corporate and university environments).

    Audio player or CD burner:  What if the user has multiple CD-ROM drives or audio systems?  I have a friend in the recording business who has three audio cards in his computer, and sometimes uses all three at once.  He often uses special audio software solely to get around the single-instance limitations that many audio/video programs have.

    Shell:  The user wants to compare different settings on the shell to see their effects, so she runs two copies, one as the "control" and one as the "experiment".

    Display properties:  I have three monitors on this very computer.  The fact that I’m only allowed one Display Properties dialog is often irritating to me, as I often want to change properties of the monitors separately — but can’t.

  10. Tim says:

    I’ve written a program that lets you keep notes on the desktop – it automatically saves periodically to a file in the user’s My Documents folder.

    I prevent multiple instances (by the same user), because otherwise data corruption could (i.e. would) occur.

    I could change the app so you can configure where to save the data, but that would make it more complicated (one of the points of the program is its simplicity), and hardly any one would use the feature.  Lots of people would be confused by it.  The user interface would be pointlessly split between multiple instances too.  There would be multiple icons in the notification area, too.  Cats and dogs living together, etc.

    I think people who can’t think of any examples of where single-instance programs are a good idea aren’t thinking all that hard.  There aren’t many, but they certainly exist.

  11. Chris Grove says:

    Sean W,

    Kiosk and other "embedded" applications are usually single-instance programs. It makes very little sense that multiple instances are available in closed or limited systems.

    In our embedded Windows system (running under Windows XP), there is a strong coupling between our hardware configuration and the user application state. There would be very little gain to allow the user an additional instance as it would have to configured identically to the original.

  12. Adam says:

    Sean: You mention a focus on usability. Do you intend your comments to also hold for system level services/daemons? e.g. the system logger, the RPC daemon, the SMB service, a threaded webserver, a database server, etc…

  13. Brian says:

    Things that run full screen (ie: video games) are almost always single instanced because triple-clicking the icon would launch two full-screen versions of the game and they’d both run super slow.  The user wouldn’t know what’s going on and think the game is just slow and buggy.

  14. A. Skrobov says:

    Task Manager, that I suspect must use a

    > desktop-local object, as even that you can

    > not bring up another instance of it on

    > another desktop using the kbd shortcut, you

    > can get another instance starting if you run

    > taskmgr.exe from e.g. a command prompt on

    > another desktop

    Task Manager must be broadcasting a secret message to all windows (on current desktop), and waiting for a response within certain time.

    When the machine is loaded too high (creating a hundred top-level windows is usually enough), you can launch any number of taskmgr’s with the keyboard shortcut.

  15. Adam says:

    One of the many ways that you hinted at that sprang to my mind here was "listening on a specific network port", as a webserver does. Only one process can listen for connections on port 80 for a given IP address. If someone else is already doing it, you don’t get to.

    Consequently, if one instance of your program has wedged, you can’t just start another one and expect it to start listening. You have to kill the old one first.

    The security priveliges that can be used in this case is to only allow specific users/executables to listen to certain ports. I know that most unix-style systems require a process to be running as root to bind to ports < 1024; ISTR something along the lines that Windows systems *can* administered to require an ACL to be set for an executable that will allow it to bind to a specific port?

  16. Coderjoe says:

    One big reason for a program being single-instance is when it uses some limited hardware resource, such as a framegrabber in a system doing manufacturing process control system. You can’t really have two programs trying to access the same card at the same time. If you tried, they would usually interfere, if the drivers even allowed you to do so.

  17. Miral says:

    Even so, the program itself shouldn’t single-instance, it should just protect the single-instance nature of the resource.

    In your example, you should be able to launch multiple instances of the process control software, provided that each one is talking to a different framegrabber.

  18. Coderjoe says:

    This process control software generally runs similar to a kiosk. It needs to start up and start running pretty much without any user intervention whatsoever, even if humans will be looking at the screen to see if there is a problem or not.

  19. David Walker says:

    Some applications that bring up modal windows for things that IMHO should not be modal — such as SQL Server 2000’s Enterprise Manager — force me to bring up two or three copies of the application.

    When I’m editing a stored procedure in Enterprise Manager, it uses a modal window to let me edit the procedure.  I often need to glance at another procedure, or look at the definition of a table, or even add a field to a table, while I’m editing the stored proc.

    It irritates me to no end that E.M. uses a modal window for something that shouldn’t be modal.

    I would be *really* annoyed if Enterprise Manager limited itself to one running copy on my system; thankfully, it doesn’t.

    (As an aside, the SQL 2005 tools are much better than the 2000 ones.)

    David

  20. steveg says:

    Is this something that could (or should have) be[en] added to the OS? Add a flag against an EXE. Single-instance executables are such a common thing to do.

    Hmmmm… it could probably be done to a "mostly good enuf" level.

    The sort of thing that needs to be avoided is an app impersonating your virus scanner or firewall.

  21. Mark Steward says:

    Norman: It’s actually the Properties page that’s modal for some obscure reason.  Try pressing Ctrl+F3.

  22. Norman Diamond says:

    > A single-instance program is its own denial

    > of service! […] Your program requirements

    > are themselves a security vulnerability.

    True and mostly false, respectively.  If the denial of service is a security vulnerability then the program requirements are a security vulnerability, but maybe that just means you have to find another way to avoid the security vulnerability.

    If your program is going to run on a Windows Mobile Smartphone or Pocket PC, then the program requirements include this denial of service.  Even though you Mr. Chen didn’t set this requirement, ordinary programmers didn’t set it either.  Your company lets you and us contend with this requirement.

    Tuesday, June 20, 2006 5:04 PM by David Walker

    > It irritates me to no end that E.M. uses a

    > modal window for something that shouldn’t be

    > modal. […] I would be *really* annoyed if

    > Enterprise Manager limited itself to one

    > running copy on my system;

    Try Outlook Express.  When you open a Message Source window it’s modal, so you have to open another running copy of Outlook Express in order to create a new mail message and switch back and forth between the Message Source of one and the text of the other, except that Outlook Express doesn’t let you open another running copy.

    Tuesday, June 20, 2006 6:26 PM by A. Skrobov

    > When the machine is loaded too high (creating

    > a hundred top-level windows is usually

    > enough), you can launch any number of

    > taskmgr’s with the keyboard shortcut.

    I didn’t need a hundres top-level windows.  Windows 2000 frequently hanged for no visible reason with only about 10 windows open but idle.  Taskmgr wouldn’t respond.  Ctrl-Alt-Delete could create another taskmgr.  After opening around 20 taskmgr windows, *those* saturated the CPU for a while, but the underlying problem didn’t go away and was still unknown.

  23. Francois Boucher says:

    We use a named mutex with the name of the EXE and also use the psapi.dll to check if the exe is really started using the full path.

    Trusting a mutex only is dangerous.

    [PSAPI doesn’t give you the path to mutex owner. How do you know that the mutex owner is another copy of yourself? -Raymond]
  24. Pascal says:

    Pieter:

    Raymond discussed why .shared sections are a security hole here: http://blogs.msdn.com/oldnewthing/archive/2004/08/04/208003.aspx

  25. pvginkel says:

    Pascal:

    Thank you. I was wondering why Raymond didn’t approve :).

  26. Dan McCarty says:

    Raymond said:
    “Your program requirements are themselves a security vulnerability.”

    So then why does the Windows shell single-instance itself?

    [Engineering is about trade-offs. In exchange for the benefits of single-instance, you become susceptible to a name squatting attack from a same-privileged user. The benefit outweighs the cost, seeing as a same-privilege user can just TerminateProcess explorer if it wants to. -Raymond]
  27. IanA says:

    Apologies, not directly a ‘A single-instance program is its own denial of service’ question, but sort of related. What Mutex namespace do applications running as Scheduled Tasks use? From experience it does not seem to be the same as Desktop Apps.

  28. Jules says:

    One of my many security mantras is "you can’t protect yourself from malware running with your own priveleges".   That applies to this situation, and to a number of others.

    For example, I never really understood the point of outbound connection filtering on a software firewall, yet I so frequently see people saying that the Windows firewall is useless because it doesn’t do it.  Err.. no, it’s not useless.  The feature’s useless, because it is trivial for any application to circumvent it if you have the privelege to use your own network connection.  Which I assume you do.

    So quit worrying about malware running in your own context.  There’s nothing you can do about it other than prevent it from getting there.

  29. Aaargh! says:

    "For example, I never really understood the point of outbound connection filtering on a software firewall"

    That’s easy. the point is to increase the visibility of the firewall software. If you sell a firewall that installs and then is never heard from again, people feel like they spent money on something that doesn’t do anything. If, however, the firewall pops up every few hours users get the impression that it’s doing a really good job protecting them.

  30. pvginkel says:

    A different reason of course is to disallow browser helper objects and zombie processes to connect to their respective servers or send their SPAM. Of course, having a computer that has these processes running is fully (brain) dead. But then again, whom are these firewalls targeted at?

  31. Mike says:

    Aaargh! wrote about outbound filtering:

    "That’s easy. the point is to increase the visibility of the firewall software."

    If you say so.

    In more security concious circles it’s however used to block traffic (-attempts) that otherwise could reach unintended or outright unwanted destinations (such as the internet), where it doesn’t belong. For examples of such traffic, one could mention both src and dst ports 135-139 and 445.

  32. Norman Diamond says:

    Incoming spams in Outlook Express used to persuade Outlook Express to call home to spam servers, informing spammers of which e-mail addresses were working and increasing the amounts of spam sent to those addresses.  Outbound filtering in firewalls stopped Outlook Express from download graphics and stopped Outlook Express from sending HTTP requests to servers other than the mail servers allowed by the users, so a lot of that feedback stopped.  XP SP2 helped stop some of this without the need for outbound firewalls any more, except where XP SP2 didn’t stop it, and except for W2K and W98 etc.

  33. Dewi Morgan says:

    The reason that outbound firewalls work at the moment (and are a very, very good thing to have) is that they are in the minority: very few users have them, and there are several available. That means it’s not worth writing malware to dodge them: you’d be writing malware for a tiny target market.

    Once Vista brings outbound filtering in, malware will need to find a way to dodge it because EVERYONE will have it. Dodging outbound filtering would have become the norm instead of the exception. So, if I understand right, the Vista outbound filtering required some core changes to how services are identified, giving each one an allegedly-unspoofable unique ID.

    Of course that’s still fairly trivial to get around for the malware, but if MS hadn’t raised the bar at least a little, then there would have been absolutely no point at all in putting outbound filtering into Vista.

    Far be it from me to point out that, with their FW doing outbound filtering, and being the only FW that protects the system at bootup, MS are looking to push the other FW producers out the market…

  34. name that is required says:

    ghm… I’m always using volatile keys under HKCU/Software instead of mutexes. Just for reason above.

Comments are closed.