Managing Your HDInsight Cluster using PowerShell – Update

Since writing my last post Managing Your HDInsight Cluster and .Net Job Submissions using PowerShell, there have been some useful modifications to the Azure PowerShell Tools. The HDInsight cmdlets no longer exist as these have now been integrated into the latest release of the Windows Azure Powershell Tools. This integration means: You don’t need to…

0

Managing Your HDInsight Cluster and .Net Job Submissions using PowerShell

This post explains how best to manage an HDInsight cluster using a management console and Windows PowerShell. The goal is to outline how to create a simple cluster, provide a mechanism for managing an elastic service, and demonstrate how to customize the cluster creation. Before provisioning a cluster one need to ensure the Azure subscription…

3

Managing Hive Job Submissions With PowerShell

In my previous post, I talked about “Managing Your HDInsight Cluster with PowerShell”. In this post I made no mention of using Hive. I hope to re-address this balance by specifically talking about how you can submit Hive jobs from the same local management console. As before all the scripts mentioned in this and the…

3

Managing Your HDInsight Cluster with PowerShell

An updated version of this post can be found here. This blog post provides a mechanism for managing an HDInsight cluster using a local management console through the use of Windows PowerShell. The goal is to outline how to configure the local management console, create a simple cluster, submit jobs using MRRunner, and finally provide…

0

Hive and XML File Processing

When I put together the “Generics based Framework for .Net Hadoop MapReduce Job Submission” code one of the goals was to support XML file processing. This was achieved by the creation of a modified Mahout document reader where one can specify the XML node to be presented for processing. But what if ones wants to…

7

.Net Hadoop MapReduce Job Framework - Revisited (Archived)

An updated version of this post can be found at: http://blogs.msdn.com/b/carlnol/archive/2012/04/29/generic-based-framework-for-net-hadoop-mapreduce-job-submission.aspx If you have been using the Framework for Composing and Submitting .Net Hadoop MapReduce Jobs you may want to download an updated version of the code: http://code.msdn.microsoft.com/Framework-for-Composing-af656ef7 The biggest change in the latest code is the modification of the serialization mechanism. Formerly data was…

0

Hadoop XML Streaming and F# MapReduce

So, to round out the Hadoop Streaming samples I thought I would put together an XML Streaming sample. As always the code can be found here: http://code.msdn.microsoft.com/Hadoop-Streaming-and-F-f2e76850 XML Streaming Reader So how does one stream in XML? If you read the Hadoop Streaming documentation you will notice the following FAQ: You can use the record…

0

Hadoop Streaming and Windows Azure Blob Storage

One of the cool features of the Microsoft Distribution of Hadoop (MDH) is the native support for Windows Azure Blob Storage. When performing HDFS operations by default one can omit the scheme such that: hadoop fs -lsr /mobile Is equivalent to: hadoop fs -lsr hdfs:///mobile The commands are defaulting to the HDFS scheme. Although Hadoop…

0

Hadoop Binary Streaming and F# MapReduce

As mentioned in my previous post Hadoop Streaming not only supports text streaming, but it also supports Binary Streaming. As such I wanted to put together a sample that supports processing Office documents; more on support for PDF in a later post. As always the code can be downloaded from: http://code.msdn.microsoft.com/Hadoop-Streaming-and-F-f2e76850 Putting together this sample…

0

MapReduce Tester: A Quick Word

In my previous post I talked a little about testing the Hadoop Streaming F# MapReduce code; but it is worth saying a few words about the tester application. The complete code for this blog post and the F# MapReduce code can be found at: http://code.msdn.microsoft.com/Hadoop-Streaming-and-F-f2e76850 As mentioned Unit Testing the individual map and Reduce functions…

0