Accessing Azure Data Lake Store using WebHDFS with OAuth2 from Spark 2.0 that is running locally

Update from March 2017: Since posting this article in August 2016, Azure Data Lake Product Team  published three new and highly recommended blog posts: Connecting your own Hadoop or Spark to Azure Data Lake Store Making Azure Data Lake Store the default file system in Hadoop Wiring your older Hadoop clusters to access Azure Data…

3

Accessing Azure Storage Blobs from Spark 1.6 that is running locally

When you are using HDInsight Hadoop or Spark clusters in Azure, they are automatically pre-configured to access Azure Storage Blobs via the hadoop-azure module that implements the standard Hadoop FilesSystem interface. You can learn more about how HDInsight uses blob storage at https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hadoop-use-blob-storage/ In this article, I will show how we can configure a local…

7

Resolving Spark 1.6.0 "java.lang.NullPointerException, not found: value sqlContext" error when running spark-shell on Windows 10 (64-bit)

It is easy to follow the instructions on http://spark.apache.org/docs/latest/ and download Spark 1.6.0 (Jan 04 2016) with the “Pre-build for Hadoop 2.6 and later” package type from http://spark.apache.org/downloads.html However, when you try to run spark-shell on your Windows 10 (64-bit) machine, you may receive a java.lang.RuntimeException: java.lang.NullPointerException (not found: value sqlContext) java.lang.RuntimeException: java.lang.NullPointerException         at…

16