Output to Azure Data Lake Store is Generally Available

We are excited to announce that the capability to output to Azure Data Lake Store, a hyper-scale repository for big data analytics workloads from Azure Stream Analytics is now Generally Available. This integration further advances the ease of enablement of a Lambda architecture where the data that is subjected to real time stream analytics is…

0

Spark Streaming and Azure Stream Analytics

As Microsoft adds support for both proprietary and open source technologies for processing and analytics of streaming data, customers have been asking us how to choose between Spark Streaming and Azure Stream Analytics. Here is my perspective on this subject, given my close involvement in the development and adoption of both these technologies. Let’s first…


Troubleshooting Azure Stream Analytics jobs with SELECT INTO

Azure Stream Analytics is a fully managed service to do real time processing of data with a flexible SQL-like language. You can easily construct queries to complete complex analysis. However, it also means sometimes the system can be hard to troubleshoot when it does not run as expected. Here is one trick that can help…


Troubleshooting Azure Stream Analytics jobs on New Portal

We are working hard to bring you an end-to-end Stream Analytics experience in the new Azure portal and we are almost there! Now you can perform more troubleshooting tasks on portal.azure.com. Input and output diagnosis: A yellow triangle warning sign on your input or output indicates something is wrong. Clicking on the warning sign will…

1

SQL Data Warehouse as output of Azure Stream Analytics

Several of our customers have asked whether we support Azure SQL Data Warehouse an output sink for of an Azure Steam Analytics job. The answer is we do: it can be configured by choosing “SQL DATABASE” output option, as shown below: The SQL Data Warehouse documentation also contains step-by-step instructions on configuring Azure SQL Data…


Integration with Azure Data Lake Store

We are excited to announce that Azure Stream Analytics can output to Azure Data Lake Store, a hyper-scale repository for big data analytics workloads. This integration further advances the ease of enablement of a Lambda architecture where the same data that is subjected to real time stream analytics is also stored and then subjected to…

0

How to configure Azure Stream Analytics outputs to skip events that cannot be written due to conversion errors or schema mismatch

By default, when output events cannot be written to the external storage due to being non-conforming data (null columns when destination column is non-nullable, longer strings than the destination column support, values with types that cannot be written in the output etc.) they cause the job to stop processing new events and continuously retry the…


How to deal with missing events in streaming data?

Streaming data is often not perfect – some of the events can be missing and some can be generated or received with delay. At the same time downstream applications may require input data within regular intervals (e.g. every 5 seconds) Some customers asked us – how can Azure Stream Analytics be used to convert a…


Handling Json array in Stream Analytics Query

This week, I am sharing a query question asked in stackoverflow to illustrate how to handle json array in Stream Analytics Query:   Problem   I retrieve some weatherdata from an external API. This is returned as JSON and send to an Azure IoT hub. Stream analytics processes the json into a proper format, but I…


Pulling data from either public or private sources to Azure Event Hub where the data is exposed as a web service or feed

In typical Internet of Things (IoT) scenarios, you have devices that you can program to push data to Azure, either to an Azure Event Hub or an IoT hub. Both of those hubs are entry points into Azure for storing, analyzing, and visualizing with a myriad of tools made available on Microsoft Azure. However, they…