Accessing Azure Data Lake Store using WebHDFS with OAuth2 from Spark 2.0 that is running locally

Update from March 2017: Since posting this article in August 2016, Azure Data Lake Product Team  published three new and highly recommended blog posts: Connecting your own Hadoop or Spark to Azure Data Lake Store Making Azure Data Lake Store the default file system in Hadoop Wiring your older Hadoop clusters to access Azure Data…

3

Compile and build specific Hadoop source code branch using Azure VM

Sometimes you may want to test a Hadoop feature that is available in a specific branch that is not available as a binary release. For example, in my case, I want to try accessing Azure Data Lake Store (ADLS) via its WebHDFS endpoint. Access to ADLS requires OAuth2, support for which was added in Hadoop…

0

Accessing Azure Storage Blobs from Spark 1.6 that is running locally

When you are using HDInsight Hadoop or Spark clusters in Azure, they are automatically pre-configured to access Azure Storage Blobs via the hadoop-azure module that implements the standard Hadoop FilesSystem interface. You can learn more about how HDInsight uses blob storage at https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-use-blob-storage/ In this article, I will show how we can configure a local…

7