SU Utilization Metric

In order to achieve low latency streaming processing, Azure Stream Analytics jobs perform all processing in memory. When running out of memory, the streaming job fails. As a result, for a production job, it’s important to monitor a streaming job’s resource usage, and make sure there is enough resource allocated, in order to keep the…


Troubleshooting Azure Stream Analytics jobs with SELECT INTO

Azure Stream Analytics is a fully managed service to do real time processing of data with a flexible SQL-like language. You can easily construct queries to complete complex analysis. However, it also means sometimes the system can be hard to troubleshoot when it does not run as expected. Here is one trick that can help…


How to deal with missing events in streaming data?

Streaming data is often not perfect – some of the events can be missing and some can be generated or received with delay. At the same time downstream applications may require input data within regular intervals (e.g. every 5 seconds) Some customers asked us – how can Azure Stream Analytics be used to convert a…


Handling Json array in Stream Analytics Query

This week, I am sharing a query question asked in stackoverflow to illustrate how to handle json array in Stream Analytics Query:   Problem   I retrieve some weatherdata from an external API. This is returned as JSON and send to an Azure IoT hub. Stream analytics processes the json into a proper format, but I…


Azure Streaming Units – how do I monitor? how do I scale?

Our CSS team does a fabulous job reaching out to us with customer questions, so we can draft out some documentation to address common questions.. Bill Carroll, Senior Escalation Engineer reached out to us today with this question. I have a customer who is monitoring the SU% as reported in management web app’s monitor widget. He…


Notify users of data received from sensors or other systems

How many times have you gone to a vending machine, just to find that the one snack bar you wanted out of the 50,000 things in the machine has run out?  If you are really hungry you could kick the machine, but well that wouldn’t help. Say you write a note to the vending machine company….


Querying JSON array with Azure Stream Analytics

Kent Weare has written a nice post on querying Json array with Azure Stream Analytics. He is getting device reads off of an Azure Event Hub. These reads are being aggregated on the publisher side and placed into a single message/event. Since the publisher is creating a message structure that contains many device reads for that specific interval Kent…


Stream Analytics query to check when no data has arrived from a given device in a configurable time window

Paolo Salvatori describes well here how you might write a stream analytics query to check when no data has arrived from a given device in a configurable time window. To solve this problem, the idea is to correlate the data stream containing real-time events (e.g. sensor readings) with device reference data. Thanks again Paolo for…