Its kind of pain to debug a debugger. One thing for sure. If you want to debug VS debugger for C++,C# etc languages then it would be bit simple(I think) as everything takes place in VS itself. So its just question of attaching to another VS which is kind of well established method here. But when it comes to debugging a script debugger it gets hard. Debugging takes place through VS but actual engine runs in IE. So just attaching to VS doesn't solve the purpose. You have to attach to IE. Now things get complicated here. Reason IE is invoked by VS which you are already debugging. So you need to attach to this IE using some other version of the VS. Basically you need unregsiter RET csm.dll and register CHK csm.dll so that you don't face problem in attaching to IE which is invoked by VS.
So the final setup looks like this:
Unregister RET csm.dll using regsvr32 /u <path for RET csm.dll>
Register CHK csm.dll using regsvr32 <path for CHK csm.dll>
CHK VS has venus project in which you can set bp in script etc. Pressing F5 invokes IE and breaks at previosly set BP. (You will need to check off option from Tools -> Internet Options -> Advanced -> Disable Script Debugging to enable script debugging)
In RET VS you need to attach to IE and CHK VS.
And you are ready to debug script debugging. Happy Debugging!!