Using Python to implement a queue publisher and subscriber with a polling back-off delay algorithm.

I recently come across Python. It’s a wonderful language with many packages for statistics, HDInsight, natural language toolkit, etc. After using Azure queue and table store separately with Python I implemented a best practice to handle large messages using table store and the queue together. The flowchart on this page shows the logic commonly used…

0

Comments on the post – Manage Hadoop clusters in HDInsight using Azure PowerShell

Overall, the post on managing a Hadoop cluster in HDInsight with Azure PowerShell was great. I altered the parts below for various reasons. My related posts can be found here.   Copying a file to/from blob store In the post linked above you’ll create a new storage container. Rather than rush ahead I wanted another way…

0

Getting familiar with Apache HBase.

This post is a first step to understanding Apache HBase. Consider this a knowledge nugget chock full O’ goodness! Specifically, I am using HDInsight which is the Microsoft Azure Hadoop distribution. My prior posts can be found here.   What is HBase and why would you use it?  HBase is one example of a “NoSQL”…

0

Getting familiar with Hive query language.

In my previous post I described Hive at a high level, the Hive query language and how Hive works with map-reduce. In this post we’ll create a hive table, get a data file and query the data in that file. These steps will also demonstrate schema on read. In addition, we’ll create a view. Lastly, we’ll…

1

An overview of Hadoop Distributed File System, HCatalog, Hive and map-reduce.

Yet another step in my journey learning about big data. Below are some things I’ve learned along the way. The image here provides a high-level overview of Hadoop. (I’m still trying to find a more detailed architecture diagram that I can freely re-use). After setting up an HDInsight cluster on Azure I dove in and wrote some…

0

Getting started with big data (HDInsight, Hadoop, etc.)

I’m currently focused on learning about big data. Over a series of posts I’ll show the path I took from being new to big data to eventually doing something worthwhile with Big Data, specifically using HDInsight which is the Microsoft Azure Hadoop distribution. If big data is new to you then I hope you can…

0