For RichEdit 4.0 (Windows XP SP1), I developed a UTF-8 version of the Rich Text Format (RTF). The reason was to have a faster, more reliable way of handling copy/paste for RichEdit than regular RTF. RichEdit 5.0 added the binary format for this purpose (and for OneNote) and RichEdit 6.0 added a still faster internal method to speed the build up of the math linear format. Accordingly starting with RichEdit 5.0 (Office 2003), the UTF-8 RTF format isn’t used for copy/paste unless the client specifically asks for it, and I hadn’t paid much attention to it.
But a UTF-8 RTF bug for the N’Ko script (the only right-to-left script that displays its digits RTL!) showed up the other day needing some attention. In my standard RTF debugging mode, I opened the file in NotePad to see what was going on. Much to my delight, it looks sooooooo much better than usual! Here’s the text part of the “new scripts for Windows 8” file written by RichEdit for standard RTF (N’Ko characters highlighted ):
You can’t tell what the characters are since they’re all represented by the RTF uN notation. Btw, this is still a lot simpler than Word writes. You can see the latter by saving a Word file in RTF and looking at the file in NotePad. The RichEdit file containing the RTF above is 1437 bytes and the corresponding Word file is 38190 bytes. You can see why people pass Word RTF files through WordPad to get something lighter.
Now here’s what the same text looks like when written in the UTF-8 RTF format
You can read all the new-script characters instead of looking at \uN control words! Well, maybe you don't understand the text, but at least you can see the characters. The file containing this RTF is 1003 bytes, about 70% the size of the RichEdit standard RTF file and about a fortieth the size of the Word RTF file.
The \uN notation is certainly very valuable, but it’s particularly awkward because it uses signed 16-bit decimal values. To find out what the characters are you have to add 65536 to negative values and convert the results to hexadecimal. Furthermore a surrogate pair is represented by two \uN control words instead of one with an unsigned integer. So you have to convert two negative 16-bit decimal numbers to hex and then convert the resulting surrogate pair to the UTF-32 form to get what’s in the Unicode Standard. Since Word writes many RTF control words with unsigned 32-bit values, there really wasn’t any reason to stick with the original signed 16-bit convention. Standard RTF writers convert characters that can be represented using a standard Windows code page to that code page, making those characters virtually unreadable unless the code page is the Western 1252 code page. Meanwhile UTF-8 RTF simply displays all characters outright. If you paste a UTF-8 RTF file into Word, you can see the characters and use the alt+x hot key to examine their values in Unicode.
Makes one think the UTF-8 RTF format is really a much better format than the original RTF format. Except that only RichEdit understands it.