How to configure Azure Stream Analytics outputs to skip events that cannot be written due to conversion errors or schema mismatch


By default, when output events cannot be written to the external storage due to being non-conforming data (null columns when destination column is non-nullable, longer strings than the destination column support, values with types that cannot be written in the output etc.) they cause the job to stop processing new events and continuously retry the operation of converting/inserting the non-conformant value.  This may not be always desirable, and thus we have recently added an option to instead drop such events.  The feature is not yet available via the portal, but can be used via Power Shell following these instructions:

  1. Use PowerShell to retrieve the job (https://msdn.microsoft.com/en-us/library/azure/mt603472.aspx)
  2. Add the optional property outputErrorPolicy with the value “drop” as in the example below
  3. Usage of outputErrorPolicy property that needs to be configured to instruct the job to drop non-conforming events at the output is shown in the example below:
  4.  Example:

{

  "location":
" West US ",
 
"properties": {
   
"sku": {

      "name": "standard"

    },

      
"eventsLateArrivalMaxDelayInSeconds": 10,      
"outputErrorPolicy": "drop",

5. Update the job using PowerShell (https://msdn.microsoft.com/en-us/library/azure/mt603479.aspx).

We are working on releasing this functionality in portal soon. In the meantime this solution can help with resolving the issue in case you come across it.

Comments (5)

  1. francop says:

    Hi
    I Have try your suggestion but the outputErrorPolicy is not stored in json definition and it no work

  2. Is this still relevant? and what is the timeline to enable this in the UI?

  3. Is this still relevant?

  4. turja says:

    I am getting this error when trying to upload the job via powershell

    New-AzureRmStreamAnalyticsJob : HTTP Status Code: BadRequest
    Error Code: InvalidRequestContent
    Error Message: The request content was invalid and could not be deserialized: ‘Could not find member
    ‘outputErrorPolicy’ on object of type ‘ResourceDefinition’. Path ‘outputErrorPolicy’, line 5, position 23.’.
    Request Id: 61db1892-7202-4161-81c2-6b0e6b34f6e3
    Timestamp (Utc):10/01/2016 15:37:22
    At line:1 char:1
    + New-AzureRmStreamAnalyticsJob -ResourceGroupName “SampleResourceGroup” -File “C: …
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : CloseError: (:) [New-AzureRmStreamAnalyticsJob], CloudException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.StreamAnalytics.NewAzureStreamAnalyticsJobCommand

    Any Idea?

  5. Joshua Chung says:

    So what happens to these messages that are dropped? Are we simply losing these messages? Is there a way to archive these failed messages separately?

Skip to main content