Rerunning many slices and activities in Azure Data Factory

Today someone asked me how to run all the data slices in their data factory on-demand in an ad-hoc fashion, to say run the whole pipeline again from scratch. For example, if you have a one-time copy data factory that is used to load a data warehouse or a development environment in its entirety, you…

0

Capture Microsoft Azure Stream Analytics logs

Microsoft Azure Stream Analytics makes building real time solution very easy. Developers can build Stream Analytics job with few clicks. While running Stream Analytics jobs, you may encounter an error which may cause Stream Analytics job to go in degrade/stop state. So it requires to figure out the error in job to troubleshoot it. There…

0

Make a custom role for users to Create an Azure Data Factory

Update – a fix is coming soon so that you do not need to use this workaround. Hope we can get Data Factory Contributors to be the minimum permissions to create a Data Factory from the Azure portal.   Azure built-in RBAC roles are pretty new and do not cover all the bases yet, so…

0

HDFS gets full in Azure HDInsight with many Hive temporary files

Sometimes when Hive is using temporary files, and a VM is restarted in an HDInsight cluster in Microsoft Azure, then those files can become orphaned and consume space. In Azure HDInsight, those temp files live in the HDFS file system, which is distributed across the local disks in the worker nodes. This is a different…

0

How to Lock a Resource Group to prevent accidental deletion of resources like HDInsight

Did you know it is possible to prevent accidental deletion of resources in Azure? This could apply to any number of resource, HDInsight, Stream Analytics jobs, Data Factories, DocumentDB accounts, etc. We can add a lock to the resource group to prevent resources from being removed inadvertantly. I found out the hard way when someone…


HDInsight Name Node can stay in Safe mode after a Scale Down

This week we worked on an HDInsight cluster where the Name Node has gone into Safe mode and didn’t leave that mode on its own. It’s not very common, but I wanted to share why it happened, and how to get out of the situation, in case it prevents a headache for someone else. HDInsight…

0

HDInsight Hive Metastore fails when the database name has dashes or hyphens

Working in Azure HDInsight support today, we see a failure when trying to run a Hive query on a freshly created HDInsight cluster. Its brand new and fails on the first try, so what could be wrong? Our Hive client app fails with this kind of error. Exception in thread “main” java.lang.RuntimeException: java.lang.RuntimeException: Unable to…

0

How to call a Azure Machine Learning Web Service from NodeJS

Azure machine learning allows data scientists and developers to embed predictive analytics into applications. To learn more about Azure machine learning visit Azure machine learning documentation . A simplified process flow for Azure machine learning is: Create an Azure machine learning workspace that has an associated Azure storage account. Login to your Azure machine learning…

0

Encoding 101 – Exporting from SQL Server into flat files, to create a Hive external table

Today in Microsoft Big Data Support we faced the issue of how to correctly move Unicode data from SQL Server into Hive via flat text files. The main issue faced was encoding special Unicode characters from the source database, such as the degree sign (Unicode 00B0) and other complex Unicode characters outside of A-Z 0-9….

0