I was able to cause this error in Internet Explorer 6 and Internet Explore 7. I found that editing my registry entry so that the user agent was greater than 260 I could cause this error in jscript dialogs. By removing entries to get below the size I could avoid the error. This is the registry location for the user agent strings (see http://support.microsoft.com/kb/218933 for setting this using IEAK):
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\User Agent\Post Platform
Calls to the navigator object method userAgent was the source of the error. I found that the size of the string to hold this value (internal to the Navigator object) is hard coded to a length of 260.
update: This affects three methods on the object: