Moving files between Azure Storage and RHEL

  • Article

written with assistance from Harshal Dharia, Michael Hansen, and Kan Mongwa of Microsoft

There are several options when you want to move files in or out of an Azure Storage account to a Red Hat Linux (RHEL) server.  Below is a quick break down of the most commonly used options.

Azure CLI

Azure CLI is design to run on Linux or Windows so is the ideal tool when a Linux machine is involved.  Azure CLI 2.0 (https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) is the latest and preferred version.  See https://docs.microsoft.com/en-us/azure/storage/common/storage-azure-cli#create-and-manage-blobs for syntax.

Example:

#Start the copy of the blob

az storage blob copy start --account-name ${storage_account} --account-key ${key} --source-account-name ${source_account} --source-account-key ${source_key} --source-container ${blob_container} --source-blob ${blob_name} --destination-container $target_container --destination-blob $target_blob_name

#Now wait while this copy happening. This could take a while.

while [ $(az storage blob show --account-name ${storage_account} --account-key ${key} --container-name $target_container --name $target_blob_name| jq .properties.copy.status| tr -d '"') == "pending" ]; do

    progress=$(az storage blob show --account-name ${storage_account} --account-key ${key} --container-name ${target_container} --name ${target_blob_name} | jq -r .properties.copy.progress)

    done_count=$(echo $progress | sed 's,\([0-9]*\)/\([0-9]*\),\1,g')

    total_count=$(echo $progress | sed 's,\([0-9]*\)/\([0-9]*\),\2,g')

    progress_percent=$((100 * $done_count / $total_count))

    echo "Copying: $progress ($progress_percent %)" && sleep 5

done

echo "Copying done"

AzCopy

AzCopy on Linux is a command-line utility designed for copying data to/from Azure Blob and File storage using simple commands.  See https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-linux?toc=%2fazure%2fstorage%2ffiles%2ftoc.json.  Specific steps for the file copy are here.

To Install on RHEL, you first need to install .NET Core 1.1.1

  1. sudo yum install rh-dotnetcore11
  2. scl enable rh-dotnetcore11 bash
  3. source scl_source enable rh-dotnetcore11

Then you install AzCopy per the article.

  1. wget -O azcopy.tar.gz https://aka.ms/downloadazcopyprlinux
  2. tar -xf azcopy.tar.gz
  3. sudo ./install.sh

Example from RHEL 7.4:

clip_image002

Wget

The final approach is to downloading is to generate a SAS token, append it to the blob URL and simply use wget in Linux.

See https://blogs.msdn.microsoft.com/nicole_welch/2017/05/using-azure-cli-1-0-to-copy-a-file-between-subscriptions/ for details on how to generate the SAS token and append it to the the blob URL (aka, "sourceSAS").