My VS 2013 would not connect to any internal or VSO TFS instances today. I also could not remove servers (??). I kept getting "Key not valid for use in specified state", not matter what I tried. This was particularly frustrating because VS 2012 on the same machine was connecting to everything just fine!
I exited VS and deleted %APPDATA%\Microsoft\VisualStudio\12.0 and %APPDATA%\Microsoft\Team Foundation\5.0 but that did not fix it. After some digging I found someone on Connect had reported same and for them deleting the 'HKEY_CURRENT_USER\Software\Microsoft\VSCommon\12.0\ClientServices\TokenStorage' folder got it working again. Sure enough, it fixed me too.
HTH if you find yourself in the same state someday.