Linux and Azure Files: you might need some help here…

  • Article

 

Hi!

tl;dr: To mount Azure Files from linux, you need cifs support in the kernel, the right mount helper and versions recent enough to supports the SMB2 protocol version.

I just got a ping from a customer who had trouble mounting an Azure Files filesystem from Linux. According to the Azure team blog, this should work: https://blogs.msdn.com/b/windowsazurestorage/archive/2014/05/12/introducing-microsoft-azure-file-service.aspx 

So I tried it myself on a Ubuntu 14.04 LTS and found the following:

If I used smbclient, everything worked:

kenn@cubefileclient:~$ smbclient -d 3 //cubefiles.file.core.windows.net/cubefiletest <storage key goes here> -U cubefiles -m SMB2
[lots of debug output deleted here]
Connecting to 168.61.61.18 at port 445
Doing spnego session setup (blob length=0)
server didn't supply a full spnego negprot
Got challenge flags:
Got NTLMSSP neg_flags=0x628a8015
NTLMSSP: Set final flags:
Got NTLMSSP neg_flags=0x60088215
NTLMSSP Sign/Seal - Initialising with flags:
Got NTLMSSP neg_flags=0x60088215
Domain=[X] OS=[] Server=[]
smb: \> dir

  .                                       D        0  Mon Sep  8 14:49:55 2014
  ..                                      D        0  Mon Sep  8 14:49:55 2014
  testdir                             D        0  Mon Sep  8 14:47:08 2014
                83886080 blocks of size 65536. 83886080 blocks available
Total bytes listed: 0
smb: \> quit

Don’t be alarmed by all those scary looking messages, I’m running smbclient with –d 3, so there are a lot of debug messages.

Now I tried to mount the filesystem:

kenn@cubefileclient:~$ sudo bash
root@cubefileclient:~# mount -t cifs \\\\cubefiles.file.core.windows.net\\cubefiletest /mountpoint -o vers=2.1,username=cubefiles,password=<storage key goes here>,dir_mode=0777,file_mode=0777
mount: wrong fs type, bad option, bad superblock on \\cubefiles.file.core.windows.net\cubefiletest,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)
       In some cases useful info is found in syslog – try
       dmesg | tail  or so

OK, this did not work.

So let’s check if the cifs filesystem is actually in the kernel.

root@cubefileclient:~# grep cifs /proc/filesystems
nodev   cifs

Yes, looks good.

So is there a mount helper for cifs?

root@cubefileclient:~# ls -la /sbin/mount.cifs
ls: cannot access /sbin/mount.cifs: No such file or directory

That’s it! we’re missing the mount helper!

root@cubefileclient:~# apt-get install cifs-utils

root@cubefileclient:~# mount -t cifs \\\\cubefiles.file.core.windows.net\\cubefiletest /mountpoint -o vers=2.1,username=cubefiles,password=<storage key goes here>,dir_mode=0777,file_mode=0777

root@cubefileclient:~# mount
[…]
\\\\cubefiles.file.core.windows.net\\cubefiletest on /mountpoint type cifs (rw)

root@cubefileclient:~# ls /mountpoint/
testdir

So this is great, and I thought this was the bug our customer was hitting. But I was wrong: Even with installing the mount helper nothing worked. Even the smbclient did not work for him.

So I recreated his setup (based on Suse Enterprise 11) and I saw the following:

cubefileclient2:~ # smbclient -d 3 //cubefiles.file.core.windows.net/cubefiletest <storage key goes here> -U cubefiles -m SMB2
[lots of debug output deleted here…]
protocol negotiation failed: NT_STATUS_PIPE_BROKEN

And also the mount failed.

So I decided to look at what’s going on on the wire. I opened up a second ssh window to the VM and ran tcpdump on the second terminal while attempting to connect to Azure Files in the first. ( tcpdump –s 65535 –w tcpdump.pcap port 445  to be precise)

Since the output of tcpdump wasn’t too enlightening, I decided to load the output using Microsoft Network Monitor and look at the packets there. (To load the capture files from tcpdump, make sure they have the extension .pcap) And then it was quite obvious:

In Ubuntu 14.04 LTS:

image

In Suse Enterprise 11:

image

The SMB2 protocol was missing. So I started looking at the version numbers of smbclient, the cifs mount helper and the kernel.

Suse:

cubefileclient2:~ # smbclient -V
Version 3.6.3-0.54.2-3282-SUSE-CODE11-x86_64
cubefileclient2:~ # uname -a
Linux cubefileclient2 3.0.101-0.35-default #1 SMP Wed Jul 9 11:43:04 UTC 2014 (c36987d) x86_64 x86_64 x86_64 GNU/Linux
cubefileclient2:~ # mount.cifs -V
mount.cifs version: 5.1

Ubuntu:
root@cubefileclient:~# smbclient -V
Version 4.1.6-Ubuntu
root@cubefileclient:~# uname -a
Linux cubefileclient 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
root@cubefileclient:~# mount.cifs -V
mount.cifs version: 6.0

So here’s the solution: The Suse Enterprise 11 images contain a cifs implementation both in the kernel and in smbclient that hasn’t the SMB2 protocol implemented yet. And Azure files requires SMB2 otherwise the protocol negotiation will fail.

One closing remark: Please check the date when this was posted, software versions change all the time and what is described here may not be accurate anymore when you read this. I’m not posting this to point to any specific bugs or to promote one distribution over the other. It’s just a fact of life that one cannot support everything with every single version of an OS or service, this post is intended to give you ideas what to look for and give you some tools to debug low-level system behavior. And of course one could have checked the version numbers first or looked for protocol version negotiation mismatches in the debug output. But when I have no clue what to look for, I found it sometimes helpful to start with the lowest level and work my way up until I find something. 

Hope this helps,
H.