How to configure Azure Stream Analytics outputs to skip events that cannot be written due to conversion errors or schema mismatch


By default, when output events cannot be written to the external storage due to being non-conforming data (null columns when destination column is non-nullable, longer strings than the destination column support, values with types that cannot be written in the output etc.) they cause the job to stop processing new events and continuously retry the operation of converting/inserting the non-conformant value.  This may not be always desirable, and thus we have recently added an option to instead drop such events.  The feature is not yet available via the portal, but can be used via Power Shell following these instructions:

  1. Use PowerShell to retrieve the job (https://msdn.microsoft.com/en-us/library/azure/mt603472.aspx)
  2. Add the optional property outputErrorPolicy with the value “drop” as in the example below
  3. Usage of outputErrorPolicy property that needs to be configured to instruct the job to drop non-conforming events at the output is shown in the example below:
  4.  Example:

{

  “location”:
” West US “,
 
“properties”: {
   
“sku”: {

      “name”: “standard”

    },

      
“eventsLateArrivalMaxDelayInSeconds”: 10,      
“outputErrorPolicy”: “drop”,

5. Update the job using PowerShell (https://msdn.microsoft.com/en-us/library/azure/mt603479.aspx).

We are working on releasing this functionality in portal soon. In the meantime this solution can help with resolving the issue in case you come across it.

Comments (0)

Skip to main content