Encryption in SMB 3.0: A protocol perspective

Encryption is one of the new SMB 3.0 security enhancements in Windows Server 2012 RTM. It can be enabled on a per-share basis, or enforced for all shares on the server. SMB 3.0 uses AES-CCM [RFC5084] as encryption algorithm, and this also provides data integrity (signing).

This blog takes a protocol walk on the topic through [MS-SMB2], and provides a sample test vector. [MS-SMB2] describes SMB 2.x and SMB 3.0 dialects. Throughout this blog, I use the terms SMB2 and SMB3 interchangeably because both share the core SMB2 Packet format. Details on encryption configuration aspects can be found in the blogs listed in references.


Capability activation


  • The client and server advertise encryption support via the SMB2_GLOBAL_CAP_ENCRYPTION capability flag during SMB 3.0 dialect negotiation. Then at least one of the following two options must be happen:
  • For enabling encryption for a whole session, SessionFlags of SessionSetup response includes SMB2_SESSION_FLAG_ENCRYPT_DATA. This is the case where encryption is enforced on the whole server.
  • For a per-share enabled encryption, ShareFlags of TreeConnect response includes SMB2_SHAREFLAG_ENCRYPT_DATA.


When encryption is active for a given exchange, it is applied before submission to the transport. An exchange could consist of a single request or response, or a series of compounded chain operations. The sender performs encryption as an outer layer after the SMB2 messages in a given submission have been assembled.


Cryptographic keys


Ko = SMB3KDF (Ki, Label, Context)

SMB3KDF() is defined as the KDF algorithm in Counter Mode, as specified in [SP800-108] section 5.1, with 'r' value of 32 and 'L' value of 128, and HMAC-SHA256 as the PRF.


Ki – Key derivation key, used as an input to the KDF. For SMB 3.0, Ki is the SessionKey.

Label – the purpose of this derived key, encoded as string and length for SMB 3.0.

Context – the context information of this derived key, encoded as string and length for SMB 3.0.

L – An integer that specifies the length of the derived keying material Ko, L is 128 bits for SMB 3.0 cryptographic keys. Note that L is a constant since all SMB 3.0 keys are 16 bytes in length (SigningKey, EncryptionKey, DecryptionKey, and ApplicationKey).

Ko – Keying material output from the KDF, a binary string of length L, where Ko is the leftmost L bits of KDF result.


  • ServerInKey  (ClientToServer) =  SMB3KDF (SessionKey, "SMB2AESCCM\0", "ServerIn \0")

EncryptionKey (Client) = DecryptionKey (Server) = ServerInKey (ClientToServer)

  • ServerOutKey (ServerToClient) = SMB3KDF ( SessionKey, "SMB2AESCCM\0", "ServerOut\0")

EncryptionKey (Server) = DecryptionKey (Client) = ServerOutKey (ServerToClient)


The server and client generate the encryption keys upon session establishment of the primary channel. If the server is configured for encryption (i.e. SmbServerConfiguration.EncryptData) and the Connection.ClientCapabilities includes the SMB2_GLOBAL_CAP_ENCRYPTION, the server generates the EncryptionKey and DecryptionKey and sets SMB2_SESSION_FLAG_ENCRYPT_DATA flag in the SessionFlags field of the SessionSetup response; the client must also generate its encryption and decryption keys.


Upon successful TreeConnect, if Session.EncryptData is FALSE and Share.EncryptData (e.g. the share is configured to require encryption), the server generates its EncryptionKey and DecryptionKey and sets the SMB2_SHAREFLAG_ENCRYPT_DATA bit in the ShareFlags field of the TreeConnect response; the client must also generate its encryption and decryption keys.


Note: These cryptographic keys are all derived from the SessionKey. As a result, SMB 3.0 signing and encryption is as secure as the session key. Not only must this key be unique and very random, but also it needs be kept secret.


Transformed message


A transformed message consists of a transform_header followed by its encrypted SMB2 message.

The transform_header has the following fields:

ProtocolId (4 bytes):  0xFD, 'S', 'M', and 'B' (in network order).

Signature (16 bytes):  Signature of the encrypted message.

Nonce (16 bytes):  An implementation-specific value that is unique for every encrypted message within a session.

OriginalMessageSize (4 bytes):  The size in bytes of the SMB2 message.

Reserved (2 bytes):  Set to zeros and ignored.

EncryptionAlgorithm (2 bytes):  the value 0x0001 is for SMB2_ENCRYPTION_AES128_CCM.

SessionId (8 bytes):  Uniquely identifies the established session for the command.


With the SMB2_ENCRYPTION_AES128_CCM algorithm, SMB 3.0 encryption implementation uses the leftmost 11 bytes of the Nonce field in the transform_header, let’s call this Aes128Nonce. Per [RFC3610], the nonce passed to AES-CCM can be between 7 and 13 bytes. Windows SMB 3.0 made the trade-off choice of 11 bytes.


Note that it would almost be impossible that all possible nonce values can be consumed on a channel, because it would take over several centuries before a sender could recycle a nonce on a given channel. 


Encrypting the message


The sender builds the transformed message with these specifics. AES-CCM [RFC5084] is called with the following inputs:

-        AES key: Session.EncryptionKey.

-        Aes-nonce: Aes128Nonce must be used.

-        Plaintext: The SMB2 message including the header and the payload.

-        The optional authenticated data (AAD):  The SMB2 transform_header excluding the ProtocolId and Signature fields; these are the 32 bytes starting from the Nonce field.


The AES-CCM outputs are:

-        Ciphertext: the encrypted SMB2 message

-        Message authentication code: the Signature field of the transform_header.


The sender appends the encrypted SMB2 message to the transform_header and sends it to the receiver.


Decrypting the message


The message is decrypted using:

-        The EncryptionAlgorithm in the transform_header.

-        The Session.DecryptionKey of the Session that corresponds to the SessionId in the transform_header.

-        The AAD passed to AES-CCM is the transform_header excluding the ProtocolId and Signature fields.

-        The nonce passed to AES-CCM is the Aes128Nonce; that is the first 11 bytes of the Nonce field.


The signature returned by the decryption algorithm is then verified against the Signature in the transform_header.


Encryption clauses


The sender encrypts the message if any of the following conditions is satisfied:

  • If the sender is sending a response to an encrypted request.
  • If Session.EncryptData is TRUE and the request or response being sent is not NEGOTIATE.
  • If Session.EncryptData is FALSE, the request or response being sent is not NEGOTIATE or SESSION_SETUP or TREE_CONNECT, and <TreeConnect|Share>.EncryptData is TRUE.


Note: TreeConnect.EncryptData is on the client side. Share.EncryptData is on the server side.


Review of encryption clauses


  • All clauses exclude any operation which does use a SessionId. A SessionId is needed to find the Session object and derive the encryption and decryption keys from its session key. For example, if the client sends a non-encrypted ECHO, Windows 8 server will respond with a non-encrypted response.
  • Clause “response to an encrypted request”: if the sender encrypts the request, the receiver will respond in the same way by encrypting the response. There is however a prerequisite that encryption is active, i.e. encryption keys have been generated. For example, if the client encrypts an ECHO, Windows 8 server responds in-kind by encrypting the response. 
  • Clause "Session.EncryptData is TRUE": SMB 3 session setup encryption goes as follows:

--         Initial session setup messages are un-encrypted as there is no session object (key etc).
--         Session binding requests must be signed, but not encrypted. Note:  Windows-based server does not encrypt session setup responses for session binding, regardless whether the client encrypts the session binding request. On the other side, Windows client does not encrypt session binding requests when Session.EncryptData is TRUE.
--         Spontaneous re-authentication of a valid session must be encrypted, otherwise the server returns STATUS_ACCESS_DENIED.
--         Re-authentication of an expired session is encrypted as well.
--         For re-authentication at reconnection after a broken connection, if the client lost all of its connections to the server, then the client would reconnect and create a new session (initial session setup). If the client lost some of its connections but not all, then the client would reconnect and bind to the existing session (session setup binding).

  • Clause “Session.EncryptData is FALSE  and <TreeConnect|Share>.EncryptData is TRUE”:

If the client performs TREE_DISCONNECT before a LOGOFF, the logoff will not be encrypted.


Compounded messages


It is expected that all operations in an encrypted message belong to the SessionId in the transform_header. The encrypted message is sent as a single submission to the underlying transport, there is no provision for a next transformed message in the transform_header.


Related compounded messages


In case of related compounded requests, all the requests/responses belong to the same SessionId. The sender indicates to the receiver that it is using the SessionId, TreeId, and FileId of the previous operation or resulted from the receiver processing that previous operation.


Unrelated compounded messages


From a broader scope, the expected protocol behaviors for sending an encrypted message can be summarized as follows, be it for compounded requests/responses or not.

The sender should not send an encrypted request separately or as part of a compounded chain (related or unrelated) that contains a SessionId different from the session used for encryption. In other words, the sender should not use a SessionId to convoy the encrypted traffic of other authenticated sessions.

The receiver should not send an encrypted response separately or as part of a compounded chain (related or unrelated) that contains a SessionId different from the session used for encryption.   Windows-based servers will respond in-kind to what the client sends. If the client violates the above statement regarding the SessionId, a Windows 8-based server will currently send the response as such.


Oplock and Lease Breaks


Oplock break notifications/acknowledgments/responses must be encrypted when encryption is active. For an Oplock, the FileID is used to derive the SessionId which is set in the notification/acknowledgement/response. See more details in MS-SMB2.

Lease break notifications - sent by the server - do not have a SessionId, and as a result, are neither signed nor encrypted.  Lease keys are not tied to a particular session from the client.

However, Lease break acknowledgements sent by the client - and their responses sent by the server - must be encrypted when encryption is active. The client is responsible for selecting a session associated with one of the existing opens associated with that Lease Key. The response is sent on the session that receives the acknowledgment.


Test vector


This sample data should be considered “as-is”. It should also be noted that examples do not replace normative protocol specifications. The reference must be [MS-SMB2].


The test program running on a Windows 8 client negotiates SMB3 and communicates with a Windows 2012 server. It opens a file and WRITEs the following content. It then READs back the file.

This is the content written and read:

Smb3 encryption testing

Hex value:



These outputs show the encryption and decryption of the WRITE and READ commands.

The decrypted content is verified to be same at the end of the SMB2 READ response.


SessionId 0x8e40014000011
SessionKey 0xB4546771B515F766A86735532DD6C4F0
SigningKey 0xF773CD23C18FD1E08EE510CADA7CF852
EncryptionKey (Client) 0x261B72350558F2E9DCF613070383EDBF
DecryptionKey (Client) 0x8FE2B57EC34D2DB5B1A9727F526BBDB5
ApplicationKey 0x77432F808CE99156B5BC6A3676D730D1


Header.Command 0x0009 WRITE
Encryption of the request ---
Key 0x261B72350558F2E9DCF613070383EDBF
AES-128-CCM nonce 0x66E69A111892584FB5ED52
SMB2 packet
transform_header.Signature 0x81A286535415445DAE393921E44FA42E
transform_header.Nonce 0x66E69A111892584FB5ED524A744DA3EE
transform_header.OriginalMessageSize 0x87
transform_header.SessionId 0x8e40014000011
Encrypted message
Transformed message
Decryption of the response ---
Key 0x8FE2B57EC34D2DB5B1A9727F526BBDB5
Transformed message
transform_header.Signature 0xA6015530A18F6D9AFFE22AFAE8E66484
transform_header.Nonce 0x86000000000000001100001400E40800
transform_header.OriginalMessageSize 0x50
transform_header.SessionId 0x8e40014000011
AES-128-CCM nonce 0x8600000000000000110000
Decrypted SMB2 packet


Header.Command 0x0008 READ
Encryption of the request ---
Key 0x261B72350558F2E9DCF613070383EDBF
AES-128-CCM nonce 0xA5123A25F983E245983F41
SMB2 packet
transform_header.Signature 0xE93601498B76D6F7A72D5EF9B6C79FAF
transform_header.Nonce 0xA5123A25F983E245983F413B8B429AF2
transform_header.OriginalMessageSize 0x71
transform_header.SessionId 0x8e40014000011
Encrypted message
Transformed message
Decryption of the response ---
Key 0x8FE2B57EC34D2DB5B1A9727F526BBDB5
Transformed message
transform_header.Signature 0xABD518B68C2F04D7879F482B689EB83F
transform_header.Nonce 0x87000000000000001100001400E40800
transform_header.OriginalMessageSize 0x67
transform_header.SessionId 0x8e40014000011
AES-128-CCM nonce  0x8700000000000000110000
Decrypted SMB2 packet




[MS-SMB2]: Server Message Block (SMB) Protocol Versions 2 and 3 Specification

[SP800-108] National Institute of Standards and Technology. "Special Publication 800-108, Recommendation for Key Derivation Using Pseudorandom Functions", October 2009, http://csrc.nist.gov/publications/nistpubs/800-108/sp800-108.pdf

[RFC5084] Housley, R., "Using AES-CCM and AES-GCM Authenticated Encryption in the Cryptographic Message Syntax (CMS)", RFC 5084, November 2007, http://www.ietf.org/rfc/rfc5084.txt

SMB 3 Security Enhancements in Windows Server 2012


Encryption in SMB3



Comments (0)

Skip to main content