Based on the msi verbose log (http://support.microsoft.com/kb/223300) I found the following root cause:
INFO : [03/03/2009 16:38:00:434] [ApplyWebFolderProperties ]: Token is ‘/LM/W3SVC/1392067981/ROOT/’.
INFO : [03/03/2009 16:38:00:434] [ApplyWebFolderProperties ]: Key path is ‘/LM/W3SVC/1392067981/ROOT/’.
INFO : [03/03/2009 16:38:00:450] [ApplyWebFolderProperties ]: Opening key ‘/LM/W3SVC/1392067981/ROOT’.
ERROR : [03/03/2009 16:38:01:513] [ApplyWebFolderProperties ]: FAILED: -2147024748
ERROR : [03/03/2009 16:38:01:528] [ApplyWebFolderProperties ]: FAILED: -2147024748
The error -2147024748 corresponds to "ERROR_PATH_BUSY". This is the code IIS is returning to the custom action when it tries to open a key. Look at http://msdn.microsoft.com/en-us/library/ms525727.aspx .This link states that the error code means, "The path specified cannot be used at this time because a handle to the key, or one of its parents or children, is already open."
You should check if you have any other programs open that are accessing the Metabase while the Custom Action runs. There might be an application that has already opened the Metabase key we were told to modify, and there’s nothing we can do until everyone else closes that key. This seems to be the most likely cause of this failure. Hence, I would suggest restarting the IIS Service (http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/003ed2fe-6339-4919-b577-6aa965994a9b.mspx?mfr=true), and also check if you have any services or scripts running that access the Metabase. So the root cause is that IIS Metabase is locked by some other application.