Azure Streaming Units – how do I monitor? how do I scale?


Our CSS team does a fabulous job reaching out to us with customer questions, so we can draft out some documentation to address common questions.. Bill Carroll, Senior Escalation Engineer reached out to us today with this question.

I have a customer who is monitoring the SU% as reported in management web app’s monitor widget. He sees that the stream will process inputs and go to SU% of 80%. The inputs will stop (no incoming events) but the SU% stays at 80% until he stops and restarts the job. He let it sit for 8 hours with no incoming events but the SU% remained at 80%. He is under the impression that the high SU% will negatively effecting future job performance.

Questions:
• Is there a description of SU% and what type of behavior a customer should expert from this metric?
• What kind of baseline SU% can be expected on a single streaming unit when no events are incoming?
• Does the SU% include the memory consumption of a container in the backed and will not decrease until garbage collection occurs in the JVM?
• How is the SU% determined?

If you are looking for an answer for these questions, please refer to our documentation here:

https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-monitoring/

80% is right at the boundary of using up all memory resources. We currently don’t aggressively reclaim memory if utilization stays under 80%. Once the utilization is over 80%, we recommend customers to add more SUs. You can use up to 6 SUs without partitioning the query. If you need more resource, we have to scale out the query, which requires user to tell us how to partition the query using the “partition by” key word, as shown here. https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-scale-jobs/

If you have such questions and would like us to build documentation, please tweet to us @AzureStreaming

 

Comments (0)

Skip to main content