Managing Your HDInsight Cluster and .Net Job Submissions using PowerShell

This post explains how best to manage an HDInsight cluster using a management console and Windows PowerShell. The goal is to outline how to create a simple cluster, provide a mechanism for managing an elastic service, and demonstrate how to customize the cluster creation. Before provisioning a cluster one need to ensure the Azure subscription…

3

Managing Hive Job Submissions With PowerShell

In my previous post, I talked about “Managing Your HDInsight Cluster with PowerShell”. In this post I made no mention of using Hive. I hope to re-address this balance by specifically talking about how you can submit Hive jobs from the same local management console. As before all the scripts mentioned in this and the…

3

Managing Your HDInsight Cluster with PowerShell

An updated version of this post can be found here. This blog post provides a mechanism for managing an HDInsight cluster using a local management console through the use of Windows PowerShell. The goal is to outline how to configure the local management console, create a simple cluster, submit jobs using MRRunner, and finally provide…

0

Submitting Hadoop MapReduce Jobs using PowerShell

As always here is a link to the “Generics based Framework for .Net Hadoop MapReduce Job Submission” code. In all the samples I have shown so far I have always used the command-line consoles. However this does not need to be the case, PowerShell can be used. The Console application which is used to submit…

0

Co-occurrence Approach to an Item Based Recommender Update

In a previous post I talked about a Co-occurrence Approach to an Item Based Recommender, that utilized the Math.Net Numerics library. Recently the Math.Net Numerics library was updated to version 2.3.0. With this version of the library I was able to update the code to more efficiently read the Sparse Matrix entries. As such I…

0

Implementing a MapReduce Join with Hadoop and the .Net Framework

I have often been asked how does one implement a Join whilst writing MapReduce code. As such, I thought it would be useful to add an additional sample demonstrating how this is achieved. There are multiple mechanisms one can employ to perform a Join operation, and the one to be discussed will be a Reduce…

1

C# MapReduce Based Co-occurrence Item Based Recommender

As promised, to conclude the Co-occurrence Approach to an Item Based Recommender posts I wanted to port the MapReduce code to C#; just for kicks and to prove the code is also easy to write in C#. For an explanation of the MapReduce post review the previous article: http://blogs.msdn.com/b/carlnol/archive/2012/07/07/mapreduce-based-co-occurrence-approach-to-an-item-based-recommender.aspx The latest version of the code…

0

MapReduce Based Co-occurrence Approach to an Item Based Recommender

In a previous post I covered the basics for a Co-occurrence Approach to an Item Based Recommender. As promised, here is the continuation of this work, an implementation of the same algorithm using MapReduce. Before reading this post it will be worth reading the Local version as it covers the sample data and general co-occurrence…

0

Framework for .Net Hadoop MapReduce Job Submission configuration update

To better support configuring the Stream environment whilst running .Net Streaming jobs I have made a change to the “Generics based Framework for .Net Hadoop MapReduce Job Submission” code. I have fixed a few bugs around setting job configuration options which were being controlled by the submission code. However, more importantly, I have added support…

0

Framework for .Net Hadoop MapReduce Job Submission Binary Output

To end the week I decided to make a minor change to the “Generics based Framework for .Net Hadoop MapReduce Job Submission”. I have been doing some work on creating a co-occurrence matrix for item recommendations. I was going to map the process to a MapReduce job(s), then came across the issue of how I…

0