Don’t Roundtrip Ciphertext Via a String Encoding


One common mistake that people make when using managed encryption classes is that they attempt to store the result of an encryption operation in a string by using one of the Encoding classes.  That seems to make sense right?  After all, Encoding.ToString() takes a byte[] and converts it to a string which is exactly what they were looking for.  The code might look something like this:



    public static string Encrypt(string data, string password)
    {
        if(String.IsNullOrEmpty(data))
            throw new ArgumentException(“No data given”);
        if(String.IsNullOrEmpty(password))
            throw new ArgumentException(“No password given”);

        // setup the encryption algorithm
        Rfc2898DeriveBytes keyGenerator = new Rfc2898DeriveBytes(password, 8);
        Rijndael aes = Rijndael.Create();
        aes.IV = keyGenerator.GetBytes(aes.BlockSize / 8);
        aes.Key = keyGenerator.GetBytes(aes.KeySize / 8);

        // encrypt the data
        byte[] rawData = Encoding.Unicode.GetBytes(data);
        using(MemoryStream memoryStream = new MemoryStream())
        using(CryptoStream cryptoStream = new CryptoStream(memoryStream, aes.CreateEncryptor(), CryptoStreamMode.Write))
        {
            memoryStream.Write(keyGenerator.Salt, 0, keyGenerator.Salt.Length);
            cryptoStream.Write(rawData, 0, rawData.Length);
            cryptoStream.Close();

            byte[] encrypted = memoryStream.ToArray();
            return Encoding.Unicode.GetString(encrypted);
        }
    }

    public static string Decrypt(string data, string password)
    {
        if(String.IsNullOrEmpty(data))
            throw new ArgumentException(“No data given”);
        if(String.IsNullOrEmpty(password))
            throw new ArgumentException(“No password given”);

        byte[] rawData = Encoding.Unicode.GetBytes(data);
        if(rawData.Length < 8)
            throw new ArgumentException(“Invalid input data”);
        
        // setup the decryption algorithm
        byte[] salt = new byte[8];
        for(int i = 0; i < salt.Length; i++)
            salt[i] = rawData[i];

        Rfc2898DeriveBytes keyGenerator = new Rfc2898DeriveBytes(password, salt);
        Rijndael aes = Rijndael.Create();
        aes.IV = keyGenerator.GetBytes(aes.BlockSize / 8);
        aes.Key = keyGenerator.GetBytes(aes.KeySize / 8);

        // decrypt the data
        using(MemoryStream memoryStream = new MemoryStream())
        using(CryptoStream cryptoStream = new CryptoStream(memoryStream, aes.CreateDecryptor(), CryptoStreamMode.Write))
        {
            cryptoStream.Write(rawData, 8, rawData.Length – 8);
            cryptoStream.Close();

            byte[] decrypted = memoryStream.ToArray();
            return Encoding.Unicode.GetString(decrypted);
        }
    }
}

The first mistake some people make is to use ASCII encoding.  This will nearly always fail to work since ASCII is a seven bit encoding, meaning any data that is stored in the most significant bit will be lost.  If your cipherdata can be guaranteed to contain only bytes with values less than 128, then its time to find a new encryption algorithm 🙂


So if we don’t use ASCII we could use UTF8 or Unicode right?  Those both use all eight bits of a byte.  In fact this approach tended to work with v1.x of the CLR.  However a problem still remains … just because these encodings use all eight bits of a byte doesn’t mean that every arbitrary sequence of bytes represents a valid character in them.  For v2.0 of the framework, the Encoding classes had some work done so that they explicitly reject illegal input sequences (As the other Shawn-with-a-w discusses here).  This leads to bad code that used to work (due to v1.1 not being very strict) to start failing on v2.0 with exceptions along the line of:



Unhandled Exception: System.Security.Cryptography.CryptographicException: Length of the data to decrypt is invalid.
   at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
   at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
   at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()

Which at first glance looks like the CryptoStream is broken.  However, take a closer look at what’s going on.  If we check the encrypted data before converting it into a string in Encrypt and compare that to the raw data after converting back from a string in Decrypt we’ll see something along the lines of:



encrypted=array [72] { 111, 49, 30, 0, 29 …. }
rawData=array [68] { 111, 49, 30, 0, 8, … }

So round tripping through the Unicode encoding caused our data to become corrupted.  That the decryption didn’t work due to having an incomplete final block is actually a blessing — the worst case scenario here is that you end up with some corrupted ciphertext that can still be decrypted — just to the wrong plaintext.  That results in your code silently working with corrupt data.


You might not see this error all the time either, sometimes you might get lucky and have some ciphertext that is actually valid in the target encoding.  However, eventually you’ll run into an error here so you should never be using the Encoding classes for this purpose.  Instead if you want to convert the ciphertext into a string, use Base64 encoding.  Replacing the two conversion lines with:



    byte[] encrypted = memoryStream.ToArray();
    return Convert.ToBase64String(encrypted);


    byte[] rawData = Convert.FromBase64String(data);

Results in code that works every time, since base 64 encoding is guaranteed to be able to accurately represent any input byte sequence.

Comments (37)

  1. Robin says:

    I have been trying this code, but it fails on the conversion of the Base64String to byte (byte[] rawData = Convert.FromBase64String. The error I got in the first place was about an invalid character, namely the comma.

    When I removed the comma and replaced it with a slash, I got the invalid length for a Base-64 char arry.

    From MSDN I learned, the string has to be at least 4 characters, ignoring the white space characters, plus it has to be a multiple of 4, ignoring the white space characters.

    Have I misread something, or am I doing something wrong here?

    I would like to know the solution.

  2. shawnfa says:

    Hi Robin,

    I haven’t seen that problem before. Do you have repro code available?

    -Shawn

  3. SKiLLa says:

    @Robin –> I tried the code above and it works fine. Did you add the:

    byte[] encrypted = memoryStream.ToArray();

    return Convert.ToBase64String(encrypted);

    byte[] rawData = Convert.FromBase64String(data);

    changes ?

  4. tyson m says:

    i cant encrypt ascii special characters like xml – how is that possible ??

  5. shawnfa says:

    The encryption classes don’t care about "special" characters, they only see a stream of bytes.  What problem are you seeing exactly?

    -Shawn

  6. Sam says:

    So I think I’m in bad shape.  My encryption function doesn’t return the same value in ASP 2.0 as it does in 1.1.  I’m sure its because of the different behavior in Encoding classes.  What can I do?  I have WAY too much data encrypted under my "bad scheme".  I need a way to get the "old encoding methods" from 1.1, and include them in my project.  Any ideas?

    Thanks a bunch.  I’m in a pickle!

    Sam

    Public Function EncryptString(ByVal Source As String) As String

    Dim larrSourceData As Byte()

    Dim larrDestinationData As Byte()

    larrSourceData = Encoding.Unicode.GetBytes(Source)

    Call SetAESValues()

    larrDestinationData = _AESManaged.CreateEncryptor.TransformFinalBlock(larrSourceData, 0, larrSourceData.Length)

    Return Encoding.Unicode.GetString(larrDestinationData)

    End Function

  7. Aleks says:

    Hi all,

    I have a problem with this code.

    When I encode two times the exactely same string I get a different encrypted string.

    Any help ?

  8. shawnfa says:

    Hi Sam,

    Your best bet is probably to bind old versions of your application to the v1.1 framework via an app.config file, and create a new version of your application which does not use the Encoding classes to store ciphertext.

    When you install the new version, you could have some sort of upgrade utility that is also bound to the v1.1 runtime and reads in the old data, writing it out in base 64.  Or you could have the new version of the application detect old data files and run the upgrade tool automatically.

    -Shawn

  9. shawnfa says:

    Hi Aleks,

    The fact that ciphertext differs does not mean that it’s incorrect.  If you’re using symmetric encryption, you should chekc that your key, IV, and padding mode are the same.  Asymmetric encryption will always have different output due to reandom padding.

    As long as you can round trip your data, you should be fine.

    -Shawn

  10. Aleks says:

    OK Now I have it functionning (I hope). It was because of the "salt" that I didn’t need.

    The problem is that, for some string, the decryption fail with the old method (without Base64) and the new one too (with Base64).

    Theses strings are passwords and I absolutely need to have it functionning as quick as possible.

    I can send some code by email I you’d like …

    Thank you shawnfa

  11. MarkW says:

    I have heard rumours that a certain type of implementation of AES (128bit) has been cracked  (in milliseconds rather than years). If this is true, how can we be sure that the Rijndael implementation within this Crypto namespace is not at risk.

    Just curious 😉

  12. shawnfa says:

    I hadn’t seen anything about AES being cracked, so I’m not sure I can comment on RijndaelManaged 🙂

    -Shawn

  13. John Glynn says:

    How can you store encrypted values in a database if they aren’t converted into strings?

  14. shawnfa says:

    Hi John,

    You could store the encrypted byte array as a blob field in the database, or you can continue to store as a string.  However, when converting to a string do not use the Encoding classes, but instead use Convert.ToBase64String / Convert.FromBase64String.  This will create a string that can always be round-tripped back to the original byte array.

    -Shawn

  15. CodeClimber says:

    Your encryption algorithm may fail moving to .NET 2.0

  16. A different, more secure, Shawn , blogged " Don’t Roundtrip Ciphertext Via a String Encoding ". I’ve

  17. Amir Hussein says:

    When I decrypt a binary file I can not be opened! The PDF Document wheb decrypted is blank. Why? amirhussein@gmail.com

  18. michelle says:

    Okaaayy… So what if we don’t want to use Base64? I’m trying to encrypt and store text that has commas, colons, etc — more than just the letters and numbers that Base64 includes.

  19. shawnfa says:

    Hi Michelle,

    Base64 is just used to encode the ciphertext, you certainly do not need to limit your input to characters that appear in the base64 set.  In fact your input to the encryption algorithm doesn’t even need to be a string at all.

    For instance (all hypothetical and not the real encodings):

    Plaintext: "Here-Is=Some:Plain, Text"

    Ciphertext: 0x12, 0x34, 0x56, 0x78, …

    Cipertext to base64: abcdefg1234==

    The in the reverse

    Base64: abcdefg1234==

    Ciphertext from base64: 0x12, 0x34, 0x56, 0x78 …

    Plaintext decrypted: "Here-Is=Some:Plain, Text"

    -Shawn

  20. Jason Siatkowski says:

    Hi All, I’m hoping that this thread still gets read. I am having a problem with encrypting and decrypting an XML file. Since the relevant code blocks are fairly short, I will post them in this message.

    This block of code passes my XML to the encryption method.

    MemoryStream myDataStream = new MemoryStream();

    myDS.WriteXml(myDataStream, XmlWriteMode.IgnoreSchema);

    byte[] myBytes = myDataStream.GetBuffer();

    blCryptography.EncryptByte(myBytes);

    string encryptedTransactionData = Convert.ToBase64String(myBytes);

    That call to blCryptography is the actual encryption method, it looks like this…

    public static byte[] EncryptByte(byte[] data)

    {

    if ( data == null || data.Length == 0 )

    throw new CryptoException("Data: Cannot use null data");

    byte[] initVectorBytes = Encoding.ASCII.GetBytes(InitVector);

    byte[] saltValueBytes  = Encoding.ASCII.GetBytes(Salt);

    SymmetricAlgorithm sma = blCryptography.CreateRijndael(PassPhrase, saltValueBytes);

    sma.IV = initVectorBytes;

    using ( MemoryStream msEncrypt = new MemoryStream() )

    using ( CryptoStream encStream = new CryptoStream(msEncrypt, sma.CreateEncryptor(), CryptoStreamMode.Write))

    {

    encStream.Write(data, 0, data.Length);

    encStream.FlushFinalBlock();

    return msEncrypt.ToArray();

    }

    }

    As far as I can tell, that works fine.

    Now, the other side is what’s giving me fits.

    Here’s the code to decrypt the xml

    string myXML = myReader.GetString(1);

    myReader.Close();

    byte[] baEncryptedData = Convert.FromBase64String(myXML);

    byte[] baClearData = blCryptography.DecryptByte(baEncryptedData);            

    MemoryStream myStream = new MemoryStream(baClearData);

    myDS.ReadXml(myStream, XmlReadMode.IgnoreSchema);

    And finally, again the call to blCryptography is the decrypt method:

    public static byte[] DecryptByte(byte[] data)

    {

    if ( data == null || data.Length == 0 )

    throw new CryptoException("Data: Cannot use null data");                  

    byte[] initVectorBytes = Encoding.ASCII.GetBytes(InitVector);

    byte[] saltValueBytes  = Encoding.ASCII.GetBytes(Salt);

    SymmetricAlgorithm sma = blCryptography.CreateRijndael(PassPhrase, saltValueBytes);

    sma.IV = initVectorBytes;

    sma.Padding = PaddingMode.None;

    using ( MemoryStream msDecrypt = new MemoryStream(data) )

    using ( CryptoStream csDecrypt = new CryptoStream(msDecrypt, sma.CreateDecryptor(), CryptoStreamMode.Read) )

    {

    // Decrypted bytes will always be less then encrypted bytes, so len of encrypted data will be big enouph for buffer.

    byte[] fromEncrypt = new byte[data.Length];                // Read as many bytes as possible.

    int read = csDecrypt.Read(fromEncrypt, 0, fromEncrypt.Length);

    if ( read < fromEncrypt.Length )

    {

    // Return a byte array of proper size.

    byte[] clearBytes = new byte[read];

    Buffer.BlockCopy(fromEncrypt, 0, clearBytes, 0, read);

    return clearBytes;

    }

    return fromEncrypt;

    }

    }

    I hope that’s readable. The only other wrinkle I can think of is that there is an image field in the XML file and it is a BASE64 encoded string. I’m wondering what happens if you base64encode a string that already is! Or vice versa!

    The problem I get is an invalid character message when I try to read the XML into the dataset.

    Can anyone help me figure this out?

    Thanks very much!

    –Jason

  21. sergeda says:

    Hi.

    Somebody managed to use base64?

    I have used:

    Dim inputInBytes() As Byte = utf8encoder.GetBytes(plainText)

    Now I have replaced it with:

    Dim inputInBytes() As Byte = Convert.FromBase64String(plainText)

    But now I’ve got another error: "Invalid length for a Base-64 char array" in this string. Please help me with this.

  22. shawnfa says:

    Hi Sergeda,

    You’ll want to use Convert.ToBase64String() here, since you’re trying to create a base64 string.

    -Shawn

  23. shawnfa says:

    Hi Jason,

    These lines of code jump out at me:

    byte[] initVectorBytes = Encoding.ASCII.GetBytes(InitVector);

    byte[] saltValueBytes  = Encoding.ASCII.GetBytes(Salt);

    are InitVector and Salt both real ASCII strings?

    -Shawn

  24. David says:

    Hmm if you call Convert.FromBase64String(plaintext) with a small string (such as "hello" I receive: "Invalid length for a Base-64 char array.".  What am I doing wrong!?

  25. shawnfa says:

    FromBase64String takes a base64 string as input, not a plaintext string.  You’re looking for ToBase64String to convert your "hello" string into base64.  (You’ll also need to convert it to a byte array — so something to the effect of Convert.ToBase64String(Encoding.UTF8.GetBytes("hello"))

  26. TheAgent says:

    Could someone please tell me if my code is suffering from the problem discussed here? I’m in a hurry and need to fix this ASAP. Here is my code:

       Public Shared Function Encrypt(ByVal text As String, Optional ByVal additionalKey As String = "") As String

           If text Is Nothing Then text = String.Empty

           tripleDes.Key = TruncateHash(additionalKey & m_key, tripleDes.KeySize 8)

           tripleDes.IV = TruncateHash("", tripleDes.BlockSize 8)

           Dim plaintextBytes() As Byte = System.Text.Encoding.Unicode.GetBytes(text)

           Dim ms As New System.IO.MemoryStream

           Dim encStream As New CryptoStream(ms, tripleDes.CreateEncryptor(), System.Security.Cryptography.CryptoStreamMode.Write)

           encStream.Write(plaintextBytes, 0, plaintextBytes.Length)

           encStream.FlushFinalBlock()

           encStream.Dispose()

           Return Convert.ToBase64String(ms.ToArray)

       End Function

       Public Shared Function Decrypt(ByVal encryptedText As String, Optional ByVal additionalKey As String = "") As String

           tripleDes.Key = TruncateHash(additionalKey & m_key, tripleDes.KeySize 8)

           tripleDes.IV = TruncateHash("", tripleDes.BlockSize 8)

           Dim encryptedBytes() As Byte = Convert.FromBase64String(encryptedtext)

           Dim ms As New System.IO.MemoryStream

           Dim decStream As New CryptoStream(ms, tripleDes.CreateDecryptor(), System.Security.Cryptography.CryptoStreamMode.Write)

           decStream.Write(encryptedBytes, 0, encryptedBytes.Length)

           Try

               decStream.FlushFinalBlock()

           Catch ex As Exception

           Finally

               decStream.Dispose()

           End Try

           Return System.Text.Encoding.Unicode.GetString(ms.ToArray) ‘Convert.ToBase64String(ms.ToArray)

       End Function

    Thank you really. I don’t have the time to read the post carefully.

  27. Odys says:

    That’s solve my problem

    Thanks!!

  28. Nidhi says:

    Hi, This is what I am using in my decrypt method..but i m getting the error of bad data Can anyone help me out:

    public static string DecryptString(string strEncData, string strKey, string strIV)

    {

    ICryptoTransform ct;

    MemoryStream ms;

    CryptoStream cs;

    byte[] byt;

    SymmetricAlgorithm mCSP=SymmetricAlgorithm.Create();

    mCSP = new TripleDESCryptoServiceProvider();

    mCSP.Key = Convert.FromBase64String(strKey);

    mCSP.IV = Convert.FromBase64String(strIV);

    ct = mCSP.CreateDecryptor(mCSP.Key,mCSP.IV);

    byt = Convert.FromBase64String(strEncData);

    ms = new MemoryStream();

    cs = new CryptoStream(ms,ct, CryptoStreamMode.Write);

    cs.Write(byt,0,byt.Length);

    cs.FlushFinalBlock();

    cs.Close();

    return Encoding.UTF8.GetString(ms.ToArray());

    }

  29. Pete says:

    I hope this gets read:

    The signature to compare is (2) concatenated base64 encoded strings with a comma delimiter between them. Client is using OsterMiller Java utilities which they claim ignores bad characters (the comma) – so when I go to decode the string to verifyData against the signature in XML, it fails. If I replace the comma with nothing it fails because of invalid characters but If I just write it to the window without the comma it has no invalid characters, just (2) paddings, which I suppose is wrong too. What do I do?

  30. Chris says:

    Maybe there is a better way, maybe I can be enlightend but this is what I came up with.

    It’s all that conversion stuff that has my head spinning.

    code:

    #Region "Security"

       Public Sub Encrypt(ByVal password As String)

           Dim s_aditionalEntropy As Byte() = CreateRandomEntropy()

           Dim secret1 As Byte() = Encoding.UTF8.GetBytes(password)

           Dim secret2 As String = Convert.ToBase64String(secret1)

           Dim secret3 As Byte() = Convert.FromBase64String(secret2)

           Dim encryptedSecret As Byte()

           ‘Encrypt the data.

           encryptedSecret = ProtectedData.Protect(secret3, s_aditionalEntropy, DataProtectionScope.CurrentUser)

           SaveSetting(TITLE, "Settings", "UserP", Convert.ToBase64String(encryptedSecret))

           SaveSetting(TITLE, "Settings", "UserE", Convert.ToBase64String(s_aditionalEntropy))

       End Sub

       Public Function Decrypt() As String

           Dim s_aditionalEntropy As Byte()

           Dim encryptedSecret As Byte()

           encryptedSecret = Convert.FromBase64String(GetSetting(TITLE, "Settings", "UserP", ""))

           s_aditionalEntropy = Convert.FromBase64String(GetSetting(TITLE, "Settings", "UserE", ""))

           If encryptedSecret.Count <> 0 Then

               Dim secret1 As Byte() = ProtectedData.Unprotect(encryptedSecret, s_aditionalEntropy, DataProtectionScope.CurrentUser)

               Dim secret2 As String = Convert.ToBase64String(secret1)

               Dim secret3 As Byte() = Convert.FromBase64String(secret2)

               Dim secret4 As String = Encoding.UTF8.GetString(secret3)

               Return secret4

           Else

               Return ""

           End If

       End Function

       Function CreateRandomEntropy() As Byte()

           ‘ Create a byte array to hold the random value.

           Dim entropy(15) As Byte

           ‘ Create a new instance of the RNGCryptoServiceProvider.

           ‘ Fill the array with a random value.

           Dim RNG As New RNGCryptoServiceProvider()

           RNG.GetBytes(entropy)

           ‘ Return the array.

           Return entropy

       End Function ‘CreateRandomEntropy

    #End Region

  31. Sven says:

    Looks interesting…

    Seems to have one more problem, cause the returned string, doesnt want to combine with my other variables.

    Would like to decode the Adress from an server like (MyServer.org)… Seems to work correctly, but if i take this returned string and try to add subfolders variables, like (ServerAdress & Folder1 & Folder2), he ignored the two variables Folder1 and Folder2… Why that ? I search for an solution about 2 days till now.. Anyone any thoughts ???

    Greets

  32. Akuma says:

    I am actually using ToBase64String and FromBase64String and still get dad data error any idea ????????????????????

    Here is the code:

    Ecrypting:  

    DESCryptoServiceProvider desCrypto = new DESCryptoServiceProvider();

    MemoryStream ms = new MemoryStream();

    CryptoStream cs = new CryptoStream(ms,desCrypto.CreateDecryptor(EncryptKey,EncryptVactor),CryptoStreamMode.Write);

    StreamWriter sw = new StreamWriter(cs);

    sw.Write(valueToEncrypt);

    ms.Flush();

    cs.Flush();                                    

    sw.Flush();

    string result = Convert.ToBase64String(ms.GetBuffer(), 0, (int)ms.Length);  

    log.Info("value To Encrypt is " + valueToEncrypt.ToString());

    log.Info("Encrypted value is " + result);

    log.Info("Encrypting successfull.");

    Decripting:

    DESCryptoServiceProvider desCrypto = new DESCryptoServiceProvider();

    byte[] buffer = Convert.FromBase64String(valueToDecrypt);  

    MemoryStream ms = new MemoryStream(buffer);

    CryptoStream cs = new CryptoStream(ms, desCrypto.CreateDecryptor(decryptKey,decryptVactor), CryptoStreamMode.Read);

    StreamReader  sw = new StreamReader(cs);                                    

    ms.Flush();

    cs.Flush();

    cs.FlushFinalBlock();

    string result = sw.ReadToEnd();

    log.Info("value To decrypt is " + valueToDecrypt.ToString());

    log.Info("Decrypted value is " + result);

    log.Info("Decrypting successfull.");

  33. shawnfa says:

    Make sure you call FlushFinalBlock when doing the encryption as well – otherwise the padding won’t get added correctly.

    -Shawn