Hadoop Configuration Files in HDInsight

 I am keeping these settings for reference purposes and use these settings for quick check.   Mapred-site.xml name value description mapred.system.dir /mapred/system No description mapred.job.tracker jobtrackerhost:9010 No description mapred.jobtracker.taskScheduler org.apache.hadoop.mapred. CapacityTaskScheduler   mapred.job.tracker.http.address jobtrackerhost:50030 No description mapred.local.dir c:\hdfs\mapred\local No description mapred.job.tracker.history.completed.location   No description mapreduce.history.server.embedded true Should job history server be embedded within Job tracker…

4

HDInsight (Hadoop on Azure) Demo: Submit MapReduce job, process result from Pig and filter final results in Hive

In this demo we will submit a WordCount map reduce job to HDInsight cluster and process the results in Pig and then filter the results in Hive by storing structured results into a table. Step 1: Submitting WordCount MapReduce Job to 4 node HDInsight cluster: c:\apps\dist\hadoop-1.1.0-SNAPSHOT\bin\hadoop.cmd jar c:\apps\Jobs\templates\635000448534317551.hadoop-examples.jar wordcount /user/admin/DaVinci.txt /user/admin/outcount The results are stored @ /user/admin/outcount…

0

Powershell script to detect the Windows Azure SDK in Azure Cloud Service VM and fix the Powershell execution issue

When you try to run Powershell commands on Windows Azure Cloud Service (PaaS) VM you may have seen exception as below: PS D:\Users\avkash> Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime Add-PSSnapin : Cannot add Windows PowerShell snap-in Microsoft.WindowsAzure.ServiceRuntime because it is already added. Verify the name of the snap-in and try again. At line:1 char:13 + Add-PSSnapin <<<< Microsoft.WindowsAzure.ServiceRuntime +…

0

Using Fiddler to decipher Windows Azure PowerShell or REST API HTTPS traffic

If you are using publishsettings with Powershell, you may not be able to decrypt HTTPS traffic. Not sure what the problem is with publishsettings based certificates however I decided to create my own certificate using MAKECERT, and use it with Powershell to get HTTPS decryption working in Fiddler. The following steps are described based on…

2

Executing Powershell commands in Windows Azure Web or Worker Role VM Instance return exception

When trying to run Windows Azure Powershell commands in Windows Azure VM (Web Role or Worker Role) instance you will hit the following exception: PS D:\Users\avkash> Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime Add-PSSnapin : Cannot add Windows PowerShell snap-in Microsoft.WindowsAzure.ServiceRuntime because it is already added. Verify the name of the snap-in and try again. At line:1 char:13 + Add-PSSnapin…

1