Collecting Business Values from AppFabric for Analytics
Synopsis: Persistence Store extensibility to store and query metadata relating to workflow instances - useful to generate IT and Business Reports from Workflow Services hosted within Windows Server AppFabric - https://msdn.microsoft.com/en-us/windowsserver/ee695849.aspx.
The ability to report Workflow Process tracking status to end users is a very important feature of any Business Process Management (BPM) product, especially ones that run tens of thousands of concurrent processes. Many ISV partners have created their own tracking service, at great cost, that tracks metadata about the Workflow Process and each activity within it. This data is then presented via an UI status dialog for a selected process instance. A typical use of the Workflow Process tracking information is presented in the graphic below.
A key feature of this tracking data is that it also includes some elements of the business data relating to the Workflow Instance. This allows ISV and Enterprise developers to provide some business contextual information along with the Workflow Process metadata. As you can see in the above status dialog the UI displays the client name and code as well as the user who initiated the process/Workflow and the users associated with completed manual tasks.
This blog post provides guidance on using Promoted Properties to generate reports that are of interest to business analysts and ITAdmins. The Promoted Properties (new in .NET4) are extensions supported by .NET4 Workflow Services hosted within AppFabric.
Scenario
Consider the typical scenario where an ISV or Enterprise IT Group develops or extends a dashboard to display the metrics around active business processes which are manifested as Workflow Services executing as an AppFabric application. In our scenario, the Workflow Service is implemented as an application that handles document format (e.g., *.docx to *.pdf) conversion. While the IT Group is interested in monitoring bottlenecks based off average processing time of the processes; the Business Analysts could be interested in the types and counts thereof. The graphic below is a typical representation of the data that addresses the needs of this scenario.
While data relating to metrics is super useful from the Business and ITAdmin perspective, it’s not available via the tracked events emitted from Workflow Services. The challenge here is to generate data relating to the state of each running Workflow Instance, capture it in the AppFabric persistence database and generate a report of it. The next few sections of this blog post will discuss a few options to capture this data and describe one preferred approach in detail.
Traditional Methods
The traditional methods to capture and report the state and other such data relating to the state of the Workflow Instances areeither Custom Tracking Records or Extracted Variables.
In the Custom Tracking Record approach, the ‘interesting’ data/record is written to the monitoring database when a Workflow Activity executes and consequently emits the custom record. In the Extracted Variable approach, the value is captured in the Monitoring database when a Workflow Activity enters a certain pre-configured state. While both are valid options, they have a common limitation – the values are emitted as part of the Workflow Service Monitoring infrastructure and therefore can be out of sync with the persisted state of a Workflow Service instance. To elaborate this limitation, consider the use case wherein after executing the Workflow Activity, the data is emitted (as a custom tracking record or triggered variable extraction). Next, the in-flight Workflow Service Instance is abandoned or aborted due to an exception which in turn causes a rollback (of the Workflow Service Instance) to the previous persistence point. Meanwhile, the state and other data relating to the ‘rolled back’ Workflow Service Instance are already written to the Monitoring database and are out of synch with the actual state of the Workflow Service Instance. Solutions using the monitoring store for state information relating to the Workflow Service Instance and will not yield accurate reports – a limitation in .NET3.5 version.
New in .NET4/Windows Server AppFabric: Promoted Properties
Promoted Properties eliminates the consistency and sync issues we encounter in the traditional approaches, because the data is written or updated directly in the persistence database when the state of the Workflow Service Instance is updated. Promoted Properties are a set of properties that are maintained at the persistence points of a Workflow Service. Via this feature- Promoted Properties, as the name implies, the Workflow Service Host can “promote” a set of data - up to 64 variables, that is of interest to the scenario. The values for these Promoted Properties are written to the System.Activities.DurableInstancing.InstancePromotedPropertiesTable and can be queried from the InstancePromotedProperties view in the AppFabric Persistence Database.
The graphic below is an example of the query results from the InstancePromotedProperties view:
The Promoted Properties structure is not normalized and has columns: Value1 through Value64, corresponding to the InstanceId and the PromotionName that labels the set of properties. No column names are stored for these values, so it is up to the consuming/reporting application to correctly interpret these columns. Furthermore, there is only one row of Promoted Properties for each Workflow Service Instance. This means that the values are inserted once and thereafter updated for every ‘persistence’ of the associated Workflow Service Instance - this is a very important point to consider while designing the reporting application, especially if data relating to previous states is important. One shortcoming of the Promoted Properties feature is that they have no representation in the extensions to the IIS Manager User Interface that Windows Server AppFabric provides – a custom application is the only way to leverage this very useful feature.
Final consideration is that the Promoted Properties are a feature of the SQL Workflow Instance Store, and so may not be available if the AppFabric installation is using a custom provider for Persistence.
Using Promoted Properties
Using Promoted Properties is quite straightforward and this section elaborates on using this very interesting feature.
Configuring the SqlWorkflowInstanceStoreBehavior
The SqlWorkflowInstanceStoreBehavior defines a Promote method through which you define the schema of the values you wish to track. This behavior must be configured when a Workflow Service Host is being loaded for the Service (e.g., before it is opened). In order to accomplish this, you need to define a custom behavior whose sole purpose is to register your Promoted Properties. The following class shows an example of such a behavior that can be modified for the specific projects needs:
1: public class PromotePropertiesBehavior : IServiceBehavior
2: {
3: XNamespace xNS = XNamespace.Get("https://samples.com/CustomProperties");
4:
5: public virtual void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters)
6: {
7: }
8:
9: public virtual void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
10: {
11: WorkflowServiceHost host = serviceHostBase as WorkflowServiceHost;
12: if (host != null)
13: {
14: SqlWorkflowInstanceStoreBehavior store = host.Description.Behaviors[typeof(SqlWorkflowInstanceStoreBehavior)] as SqlWorkflowInstanceStoreBehavior;
15:
16: if (store != null)
17: {
18: List<;XName> variantProperties = new List<XName>()
19: {
20: xNS.GetName("ConvertFrom"),
21: xNS.GetName("ConvertTo"),
22: xNS.GetName("StartOrComplete"),
23: xNS.GetName("LastPropUpdateDateTime"),
24: xNS.GetName("Duration")
25: };
26:
27: store.Promote("ConversionStats", variantProperties, null);
28:
29: host.WorkflowExtensions.Add<;CustomPropertiesParticipant>(() => new CustomPropertiesParticipant());
30: }
31: }
32: }
33:
34: public virtual void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase)
35: {
36: }
37:
38: }
In order to register this behavior with a Service hosted in AppFabric, you will want to be able to declare it in the web.config by defining the BehaviorExtensionElement. The following code snippet shows how you can do this for the previously supplied Service behavior:
1: public class PromotePropertiesBehaviorElement : BehaviorExtensionElement
2: {
3: private ConfigurationPropertyCollection properties;
4:
5: protected override object CreateBehavior()
6: {
7: PromotePropertiesBehavior behavior = new PromotePropertiesBehavior();
8: return behavior;
9: }
10:
11: public override Type BehaviorType
12: {
13: get
14: {
15: return typeof(PromotePropertiesBehavior);
16: }
17: }
18:
19: protected override ConfigurationPropertyCollection Properties
20: {
21: get
22: {
23: if (this.properties == null)
24: {
25: ConfigurationPropertyCollection props = new ConfigurationPropertyCollection();
26: this.properties = props;
27: }
28: return this.properties;
29: }
30: }
31: }
32:
Define Persistence Participant
To ensure Promoted Properties are updated when the state of the Workflow Service Instance is updated in the Persistence Store, we need to define a PersistenceParicipant that can correctly map values supplied from a Workflow Service Instance to the schema defined in the Service behavior. The following class shows how we might do this for this blog’s scenario. Of course, this needs to be appropriately modified to be used in your scenario.
1: public class CustomPropertiesParticipant: PersistenceParticipant
2: {
3: XNamespace xNS = XNamespace.Get("https://samples.com/CustomProperties");
4:
5: public string ConvertFrom;
6: public string ConvertTo;
7: public string StartOrComplete;
8: public DateTime LastPropUpdateDateTime;
9: public double Duration;
10:
11: protected override void CollectValues(out IDictionary<;System.Xml.Linq.XName, object> readWriteValues, out IDictionary<System.Xml.Linq.XName, object> writeOnlyValues)
12: {
13: base.CollectValues(out readWriteValues, out writeOnlyValues);
14:
15: readWriteValues = new Dictionary<XName, object>();
16: readWriteValues.Add(xNS.GetName("ConvertFrom"), this.ConvertFrom);
17: readWriteValues.Add(xNS.GetName("ConvertTo"), this.ConvertTo);
18: readWriteValues.Add(xNS.GetName("StartOrComplete"), this.StartOrComplete);
19: readWriteValues.Add(xNS.GetName("LastPropUpdateDateTime"), this.LastPropUpdateDateTime);
20: readWriteValues.Add(xNS.GetName("Duration"), this.Duration);
21:
22: writeOnlyValues = null;
23:
24: }
25:
26: protected override void PublishValues(IDictionary<;System.Xml.Linq.XName, object> readWriteValues)
27: {
28: base.PublishValues(readWriteValues);
29:
30: try
31: {
32:
33: this.ConvertFrom = readWriteValues[xNS.GetName("ConvertFrom")] as string;
34: this.ConvertTo = readWriteValues[xNS.GetName("ConvertTo")] as string;
35: this.StartOrComplete = readWriteValues[xNS.GetName("StartOrComplete")] as string;
36:
37: if (readWriteValues[xNS.GetName("LastPropUpdateDateTime")] != null)
38: {
39: this.LastPropUpdateDateTime = (DateTime)readWriteValues[xNS.GetName("LastPropUpdateDateTime")];
40: }
41:
42: if (readWriteValues[xNS.GetName("Duration")] != null)
43: {
44: this.Duration = (double)readWriteValues[xNS.GetName("Duration")];
45: }
46:
47: }
48: catch (Exception ex)
49: {
50:
51: }
52:
53: }
54:
55: protected override IDictionary<;System.Xml.Linq.XName, object> MapValues(IDictionary<System.Xml.Linq.XName, object> readWriteValues, IDictionary<System.Xml.Linq.XName, object> writeOnlyValues)
56: {
57: return base.MapValues(readWriteValues, writeOnlyValues);
58: }
59:
60: }
61:
In the above code snippet, the CollectValues method is what maps the values supplied from the Workflow Service Instance to the schema when the Workflow is persisted. The PublishValues method loads the values read from Persistence and makes them available to your instance for reading and writing.
Web.config for the Scenario
With the appropriate classes in place, you finally need to add the behavior to the web.config of the Service. Below is the complete web.config for the scenario.
1: <?xml version="1.0" encoding="utf-8"?>
2: <configuration>
3: <system.web>
4: <compilation debug="true" targetFramework="4.0" />
5: </system.web>
6: <system.serviceModel>
7: <behaviors>
8: <serviceBehaviors>
9: <behavior>
10: <serviceMetadata httpGetEnabled="true" />
11: <serviceDebug includeExceptionDetailInFaults="true" />
12: <promoteProperties />
13: </behavior>
14: </serviceBehaviors>
15: </behaviors>
16: <extensions>
17: <behaviorExtensions>
18: <add name="promoteProperties" type="ServiceExtensions.PromotePropertiesBehaviorElement, ServiceExtensions, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" />
19: </behaviorExtensions>
20: </extensions>
21: </system.serviceModel>
22: <system.webServer>
23: <modules runAllManagedModulesForAllRequests="true" />
24: </system.webServer>
25: </configuration>
MWA Schema to include promotedProperties Element
First step is to register the behavior extension which adds the type of the Custom Behavior Element and provides a name for how the element will appear in the config file. Second, add the Custom Behavior to the Service’s collection of Service behaviors. Both are included in the config file above.
While we have defined a custom schema element in the config file, the Microsoft.Web.Administration (MWA) classes that AppFabric uses to read web.config will not recognize the Custom Service Behavior tag (e.g., in this case they will not recognize the promoteProperties element). This will result in an error on the IIS Manager dashboard. This is resolved by updating the MWA the schema to include the custom element. The updated schema is loaded in folder <drive>:\windows\system32\inetsrv\config\schema.
This schema (ServiceExtensions_Schema.xml) includes the <promotedProperties> element within the serviceBehaviors element in system.serviceModel/behavior node. The appropriate snippet of the schema is provided below.
1: <;configSchema>
2: <;sectionSchema name="system.serviceModel/behaviors">
3: <;element name="serviceBehaviors">
4: <;collection addElement="behavior" removeElement="remove" clearElement="clear" allowDuplicates="true">
5: <;element name="promoteProperties"/>
6: <;/collection>
7: <;/element>
8: <;/sectionSchema>
9: <;/configSchema>
Using PromotedProperties in Workflow Services
With the infrastructure in place, the Workflow Service Instance is ready to generate the Promoted Properties. Technically, the Workflow Activities are responsible for updating the values as a part of their execution. For convenience, below is a code snippet of a Custom Workflow Activity whose sole purpose is to update these Promoted Property values when it executes.
1: public sealed class CaptureAnalyticProperty : CodeActivity
2: {
3: public InArgument<;string> ConvertFrom { get; set; }
4: public InArgument<;string> ConvertTo { get; set; }
5: public InArgument<;string> StartOrComplete { get; set; }
6:
7: protected override void Execute(CodeActivityContext context)
8: {
9: string valConvertFrom = context.GetValue(this.ConvertFrom);
10: string valConvertTo = context.GetValue(this.ConvertTo);
11: string valStartOrComplete = context.GetValue(this.StartOrComplete);
12:
13: try
14: {
15: context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().ConvertFrom = valConvertFrom;
16: context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().ConvertTo = valConvertTo;
17: context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().StartOrComplete = valStartOrComplete;
18:
19: DateTime currentTime = DateTime.UtcNow;
20:
21: if (valStartOrComplete.ToLower().Equals("complete"))
22: {
23: context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().Duration =
24: (currentTime - context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().LastPropUpdateDateTime).TotalSeconds;
25: }
26:
27: context.GetExtension<;ServiceExtensions.CustomPropertiesParticipant>().LastPropUpdateDateTime = currentTime;
28: }
29: catch (Exception ex)
30: {
31: CustomTrackingRecord rec = new System.Activities.Tracking.CustomTrackingRecord("CaptureAnalyticPropertyError", System.Diagnostics.TraceLevel.Error);
32: rec.Data.Add("Suggested Fix:", "Verify that the PromotePropertiesBehavior has been added in config.");
33: rec.Data.Add("Error:", ex.ToString());
34: context.Track(rec);
35: }
36:
37: }
38: }
To generate Promoted Properties we simply drag this Custom Workflow Activity into the Workflow design and provide its arguments the appropriate values. During the Workflow execution and persistence, the values are updated by this activity in the InstancePromotedPropertiesTable and its corresponding view, InstancePromotedProperties.
Real world implementations
In this section i reference a real world solution implemented by an ISV using this Promoted Properties feature. The ISV Solution at any give time have several instances of the same long running Workflow, processing a financial transaction for example waiting for payment authorization. The Promoted Properties feature prevented the ISV from having to store metadata along with the WorkflowInstanceId that would be needed to query against. This functionality also provided the additional extensibility option to use the eventing model to resume a particular WorkflowInstance based upon an event scenario and selecting the proper WorkflowInstance to resume based on querying the metadata.
Trending the metadata is also an interesting by product. The ISV archive the data rather than just deleting it. The ISV aligns the metadata with performance monitoring data, and looks at this historically to determine the user preference and how often poor performing Workflows are being invoked or particular parameter/metadata sets that are causing typically well performing workflows to have a dramatic change.
Additional Resources
Windows Server AppFabric: https://msdn.microsoft.com/en-us/windowsserver/ee695849.aspx
MSDN Documentation on Promoted Properties - https://msdn.microsoft.com/en-us/library/ee364726(v=VS.100).aspx