Hadoop .Net HDFS File Access

Provided with the Microsoft Distribution of Hadoop, HDInsight, is a C library for HDFS file access. This code extends this library through a Managed C++ solution. This solution enables one to consume HDFS files from within a .Net environment. The purpose of this post is first to ensure folks know about the new Windows HDFS…

8

Submitting Hadoop MapReduce Jobs using PowerShell

As always here is a link to the “Generics based Framework for .Net Hadoop MapReduce Job Submission” code. In all the samples I have shown so far I have always used the command-line consoles. However this does not need to be the case, PowerShell can be used. The Console application which is used to submit…

0

Co-occurrence Approach to an Item Based Recommender Update

In a previous post I talked about a Co-occurrence Approach to an Item Based Recommender, that utilized the Math.Net Numerics library. Recently the Math.Net Numerics library was updated to version 2.3.0. With this version of the library I was able to update the code to more efficiently read the Sparse Matrix entries. As such I…

0

Execution Time Based Heuristic Custom Task Scheduler

If you follow the samples for Parallel Programming with the .Net Framework, you may have come across the ParallelExtensionsExtras and the Additional TaskSchedulers. Although these samples cover a broad set of requirements I recently came across another that could be satisfied with the creation of a new custom task scheduler. In the current samples there…

0

Framework for .Net Hadoop MapReduce Job Submission TextOutput Type

Some recent changes made to the “Generics based Framework for .Net Hadoop MapReduce Job Submission” code were to support Json and Binary Serialization from Mapper, in and out of Combiners, and out from the Reducer. However, this precluded one from controlling the format of the Text output. Say one wanted to create a tab delimited…

0

C# MapReduce Based Co-occurrence Item Based Recommender

As promised, to conclude the Co-occurrence Approach to an Item Based Recommender posts I wanted to port the MapReduce code to C#; just for kicks and to prove the code is also easy to write in C#. For an explanation of the MapReduce post review the previous article: http://blogs.msdn.com/b/carlnol/archive/2012/07/07/mapreduce-based-co-occurrence-approach-to-an-item-based-recommender.aspx The latest version of the code…

0

MapReduce Based Co-occurrence Approach to an Item Based Recommender

In a previous post I covered the basics for a Co-occurrence Approach to an Item Based Recommender. As promised, here is the continuation of this work, an implementation of the same algorithm using MapReduce. Before reading this post it will be worth reading the Local version as it covers the sample data and general co-occurrence…

0

Co-occurrence Approach to an Item Based Recommender

For a while I thought I would tackle the problem of creating an item-based recommender. Firstly I will start with a local variant before moving onto a MapReduce version. The current version of the code can be found at: http://code.msdn.microsoft.com/Co-occurrence-Approach-to-57027db7 The approach taken for the item-based recommender will be to define a co-occurrence matrix based…

0

Framework for .Net Hadoop MapReduce Job Submission Binary Output

To end the week I decided to make a minor change to the “Generics based Framework for .Net Hadoop MapReduce Job Submission”. I have been doing some work on creating a co-occurrence matrix for item recommendations. I was going to map the process to a MapReduce job(s), then came across the issue of how I…

0

Framework for .Net Hadoop MapReduce Job Submission libjars update

If you have been using the “Generics based Framework for .Net Hadoop MapReduce Job Submission” you may want to download the latest version of the code. The previous version of the code, when processing XML and Binary files, was dependent on a custom streaming JAR that contained the necessary reader classes. This was not an…

0