Using Azure Stream Analytics JavaScript UDF to lookup values in JSON array

JavaScript UDF (User-Defined Function) allows you to handle complex JSON schema and keep your query clean. In this blog you will learn how to handle nested JSON arrays with a JavaScript UDF. Below is an example event generated by an IoT gateway. There is an array of two devices – device01 and device02; each device…

0

Processing Configurable Threshold Based Rules in Azure Stream Analytics

This post covers the usage of Azure Stream Analytics to process configurable threshold based rules. Canonical scenarios where an alert is to be generated when an event with a certain value occurs or when an aggregated value exceeds a certain threshold can be articulated as threshold based rules in Azure Stream Analytics. These queries are…


How to achieve exactly-once delivery for SQL output

Azure Stream Analytics guarantees exactly-once processing within the processing pipeline. However, it currently doesn’t ensure end to end exactly-once delivery to output sink. Instead, it guarantees at-least-once delivery to output sink. When using SQL output, we can achieve exactly-once semantics if the following requirements are met: all output streaming events have a natural key, i.e….


Output to Azure Data Lake Store is Generally Available

We are excited to announce that the capability to output to Azure Data Lake Store, a hyper-scale repository for big data analytics workloads from Azure Stream Analytics is now Generally Available. This integration further advances the ease of enablement of a Lambda architecture where the data that is subjected to real time stream analytics is…

0

Spark Streaming and Azure Stream Analytics

As Microsoft adds support for both proprietary and open source technologies for processing and analytics of streaming data, customers have been asking us how to choose between Spark Streaming and Azure Stream Analytics. Here is my perspective on this subject, given my close involvement in the development and adoption of both these technologies. Let’s first…


Troubleshooting Azure Stream Analytics jobs with SELECT INTO

Azure Stream Analytics is a fully managed service to do real time processing of data with a flexible SQL-like language. You can easily construct queries to complete complex analysis. However, it also means sometimes the system can be hard to troubleshoot when it does not run as expected. Here is one trick that can help…


Troubleshooting Azure Stream Analytics jobs on New Portal

We are working hard to bring you an end-to-end Stream Analytics experience in the new Azure portal and we are almost there! Now you can perform more troubleshooting tasks on portal.azure.com. Input and output diagnosis: A yellow triangle warning sign on your input or output indicates something is wrong. Clicking on the warning sign will…

0

SQL Data Warehouse as output of Azure Stream Analytics

Several of our customers have asked whether we support Azure SQL Data Warehouse an output sink for of an Azure Steam Analytics job. The answer is we do: it can be configured by choosing “SQL DATABASE” output option, as shown below: The SQL Data Warehouse documentation also contains step-by-step instructions on configuring Azure SQL Data…


Integration with Azure Data Lake Store

We are excited to announce that Azure Stream Analytics can output to Azure Data Lake Store, a hyper-scale repository for big data analytics workloads. This integration further advances the ease of enablement of a Lambda architecture where the same data that is subjected to real time stream analytics is also stored and then subjected to…

0

How to configure Azure Stream Analytics outputs to skip events that cannot be written due to conversion errors or schema mismatch

By default, when output events cannot be written to the external storage due to being non-conforming data (null columns when destination column is non-nullable, longer strings than the destination column support, values with types that cannot be written in the output etc.) they cause the job to stop processing new events and continuously retry the…