Learn more about Azure Stream Analytics Time Skew Policies


In Stream Analytics, all data stream events have a timestamp associated with them. As all events are temporal in nature and timing of arrival of the event is how the timestamp is assigned, considerations exists for both the tolerance of out of order events and the late arrival of events to the Stream Analytics job. Contributors to Late Arrival and Out of Order event vary but generally are one or more of the following:

•Producers of the events have clock skews. This is common when producers are from different machines, so they have different clocks.

•Network delay from the producers sending the events to Event Hub.

•Clock skews between Event Hub partitions. This is also a factor because we first sort events from all Event Hub partitions by event enqueue time, and then examine the disordness.

Read about Azure Stream Analytics Time Skew Policies here

 

Comments (2)

  1. Richard Chien says:

    Could you explain more on how late arrival window contributes to latency? From what I gather it seems to depend on number of partitions in input as well? So the group by window duration (tumbling window and etc) does not matter?

    From my experiment it doesn't seem to be a linear increase. Meaning if late arrival window is 7 days it delays the output by 2 hours. I'm puzzled as to the why that is the case. It would be great if ASA team can share some insights on how large late arrival window impacts latency.

  2. Late arrival policy is not affecting latency if your inputs are arriving in timely manner across all input partitions. However, if one of the partitions stops receiving data, ASA will wait for it up to the time limit specified in late arrival policy. If there is still no event on that partition, ASA will assume that there is no data and will move time forward. So in the worst case you can see delay up to the value specified in late arrival policy.

Skip to main content