Model Driven Content Based Routing using SQL Server Modeling CTP – Part II

Author: Dana Kaufman

Reviewers: Jaime Alva Bravo, Chris Sells, Kraig Brockschmidt, Matthew Snider, David Miller

Introduction

The RouterManager sample uses the recently released SQL Server Modeling CTP and the new WCF Routing Service in .NET 4 to show how to do model driven content based message routing. Part I of the series Model Driven Content Based Routing using SQL Server Modeling CTP covered the overall architecture of the sample application. Part II will drill into the domain-specific language (DSL) developed using the M language features which makes it easy to configure Routing Service routing rules.

The routing language in the RouterManager sample consists of easy to understand, human readable text. The example listing below shows a router configuration implemented in a router DSL:

 router begin
  if header version = "Precise" priority 0 destination https://localhost:9090/servicemodelsamples/regularcalc
  if header version = "Rounded" destination https://localhost:9091/servicemodelsamples/roundingcalc
  if endpoint = "ServiceA" destination https://localhost:9090/servicemodelsamples/regularcalc, https://localhost:9091/servicemodelsamples/roundingcalc
end

Listing 1

As seen in the listing above, a router configuration starts with the text router begin and end with end. Router configurations consist of a set of filters which are rules that tell the Routing Service what content to look for in the incoming message and where matching messages should be sent. The router language uses if statements to specify a filter. After the if  keyword come the filter type. The RouterManager host supports the following filter types:

The if statement has specific syntax for each filter. For instance the header filter shown above requires a field to be specified so that the Routing Service knows what SOAP header to look for in the incoming message. A value is also required on most of the filters ( = “{value}”  ) to tell the Routing Service which pieces of data needs to be match for the filter rule to be true. Also, each rule filter can have an optional priority value which determines the execution order of the filters. Lastly, each filter can have one or more destinations to which the message can be routed.

Domain-Specific Language for Router Configuration

The “M” language features of the SQL Server Modeling CTP can be used to project arbitrary textual data into a data model. In the RouterManager sample, the language features are used to parse the configuration text and turn it into data which can be stored in the Repository database. The RouterManager sample contains a file called RouterGrammar.mg which defines a language called RouterConfigure that tells the “M” language compiler how to parse the above configuration text. The RouterGrammar file along with the model for the Router configuration can be found in the RouterModel project in the RouterManager Visual Studio solution. The CTP contains a handy tool for developing languages called “Intellipad”, a screenshot of the RouterGrammar.mg file in “Intellipad” is shown below:

 
Image 1

In the above picture, “Intellipad” is in a special language development mode. To get it into this mode, create or load a .mg language file then select “DSL” in the “Intillipad” menu and then choose “Open File As Input and Show Output… ” which will bring up a file open dialog. Select a text file with an .mg extension as input for the language and “Intellipad” will open the windows shown above. The input text is shown in the right window, the center window contains the language definition and the right window shows the parsed output. The window on the bottom shows any errors.

The RouterConfigure language is contained in a module called RouterGrammar. A grammar is a set of rules which determine if sequences of characters conform to a language. The grammar parser will parse the input text based on the rules of the grammar and build a syntax tree which is the resulting hierarchy of data that was parsed. This syntax tree can be used in a number of ways in an application including storing the parsed data (e.g. in the SQL Server Modeling Services Repository or as XML) or iterating through the tree in memory at runtime directly in your application. The RouterManager sample uses the “M” tool chain to generate the syntax tree as a set of model values that get imported into the SQL Server Modeling Services Repository.

Below is the full listing of the RouterConfigure language grammar:

    1: module RouterGrammar{
    2:   language RouterConfigure{
    3:   syntax Main = TkStart s:Statement* TkEnd => RouterRules{ valuesof(s)};
    4:   
    5:   syntax Statement = TkIf a:FilterType  => {Filter => a};
    6:  
    7:   syntax FilterType = h:Header  => h | 
    8:                       ap:AddressPrefix => ap | 
    9:                       ad:Address=>ad | 
   10:                       ac:Action => ac | 
   11:                       ma:MatchAll => ma |
   12:                       ep:Endpoint => ep | 
   13:                       aa:And =>aa | 
   14:                       c:Custom => c;
   15:                       
   16:   syntax Header = a:TkHeader f:TkField o:TkOperator v:TkValue p:Priority TkTarget t:Destinations(TkDestination,",") 
   17:    => {action => a, field=>f,operator=>o, fieldvalue=>v, priority=>p, Destinations => t};
   18:   syntax Action = a:TkAction o:TkOperator v:TkValue p:Priority TkTarget t:Destinations(TkDestination,",")
   19:     => {action => a, operator=>o, field=>"",fieldvalue=>v, priority=>p, Destinations => t};
   20:   syntax AddressPrefix = a:TkAddressPrefix o:TkEquals v:TkValue p:Priority TkTarget t:Destinations(TkDestination,",")
   21:     => {action => a, operator=>o, field=>"", fieldvalue=>v, priority=>p, Destinations => t};
   22:   syntax Address = a:TkAddress o:TkEquals v:TkValue p:Priority TkTarget t:Destinations(TkDestination,",")
   23:     => {action => a, operator=>o, field=>"", fieldvalue=>v, priority=>p, Destinations => t};
   24:   syntax Endpoint = a:TkEndpoint o:TkEquals v:TkValue p:Priority TkTarget t:Destinations(TkDestination,",")
   25:     => {action => a, operator=>o, field=>"", fieldvalue=>v, priority=>p, Destinations => t};
   26:   syntax And = a:TkAnd a1:TKAndText p:Priority TkTarget t:Destinations(TkDestination,",")
   27:     => {action => a, operator=> "",field=>"",fieldvalue=>a1, priority=>p, Destinations => t};
   28:   syntax MatchAll = a:TkMatchAll p:Priority TkTarget t:Destinations(TkDestination,",")
   29:     => {action => a, operator=>"", field=>"",fieldvalue=>"", priority=>p, Destinations => t};
   30:   syntax Custom = a:TkCustom f:TKCustomType v:TkValue p:Priority  TkTarget t:Destinations(TkDestination,",")
   31:     => {action => a, operator=>"", field=>f, fieldvalue=>v, priority=>p, Destinations => t};
   32:   syntax Priority = TkPriority n:TkNumber => n  | empty => "";
   33:   syntax Destinations(dest, separator)  = n:dest=> {{URL => n}} | l:Destinations(dest, separator) separator n:dest => { valuesof(l), {URL => n} };
   34:  
   35:   
   36:   token TkStart = "router begin";
   37:   token TkEnd = "end";
   38:   token TkName = TkText;
   39:   token TkIf = "if";
   40:   token TkHeader = "header";
   41:   token TkAction = "action";
   42:   token TkAddressPrefix = "addressprefix";
   43:   token TkAddress = "address";
   44:   token TkAnd = "and";
   45:   token TkMatchAll = "matchall";
   46:   token TkEndpoint = "endpoint";
   47:   token TkCustom = "custom";
   48:   token TkPriority ="priority";
   49:  
   50:   token TkField = TkText;
   51:   token TkOperator = TkEquals | "contains" | "startswith" | "endswith";
   52:   token TkEquals = "=";
   53:   token TkValue = '"' (any - '"' )* '"';
   54:   token TkTarget = "destination";
   55:   token TKAndText = ("A".."Z" | "a".."z" | "0".."9" | "_" | ",")+;
   56:   token TKCustomType = ("A".."Z" | "a".."z" | "0".."9" | "_" | ".")+;
   57:   
   58:   token TkDestination = TkUrl; 
   59:   token TkUrl = ("A".."Z" | "a".."z" | "0".."9" | "_" | "/" | ":" | "."  )+;
   60:  
   61:   token TkText = ("A".."Z" | "a".."z" | "0".."9" | "_")+;
   62:   token TkNumber =("0".."9")+;
   63:   
   64:   interleave Whitespace = ' ' | '\t' | '\n' | '\r';
   65:   }
   66: }

Listing 2

In a language definition, the interleave indicates the rule for which values should be used as whitespace. For the RouterConfigure language, whitespace will be defined as spaces, tabs, carriage returns/linefeed, and commas. The interleave command (line 64) looks like this:

interleave Whitespace = ' ' | '\t' | '\n' | '\r' | ',';

Tokens are used to designate rules that define the language. The language parser will try to match the tokens in the input text. Lines 36 to 62 define the tokens that make up the RouterConfigure language. For example, the token that signifies the beginning of a router configuration is called TkStart, the TkStart token tells the language parser to look specifically for the text “router begin”:

token TkStart = "router begin";

Another token TkNumber (line 62) is used for integer values. The definition for this token indicates that a number is made up of digits between “0” and “9” (the .. means between) and the “+” sign in the definition indicates a number can be made up of more than one of these characters:

token TkNumber = (“0”..”9”)+;

A token for text (line 61) is defined as one or more characters which can be any letter between A and Z (lower or upper case), any digit between 0 and 9, or an underscore. The definition for the text token looks like this:

token TkText = ("A".."Z" | "a".."z" | "0".."9" | "_")+;

And a field (TkField line 50) and a name (TkName line 38) tokens are made up of a TkText token.

There are also tokens defined for each of the filter rules (lines 40 to 48). Other tokens define information needed for specific filters and other keywords such the token TkTarget (line 54) which specifies the keyword that indicates a URL or set of URLs will follow.

Next the syntax defines how the tokens should be used (lines 3 to 33). The “M” language below is the syntax statement that tells how the tokens can be used to parse the input text. All “M” language files must have a Main syntax statement which is the starting point for the grammar rules. The ConfigRouter main syntax looks like this (line 3):

syntax Main = TkStart s:Statement* TkEnd => RouterRules{ valuesof(s)};

The above indicates that first; there should be a TkStart token in the text, then one or more Statements, which is also a syntax definition and then a TkEnd token. The values of tokens or syntax evaluations can be referenced by prefacing the token with a <name>:. The s: is used to reference the value of each statement.

Projections specify how the values parsed from the language input text should appear in the output values. The projection operator => is used and immediately followed by the pattern we want to use, in place of the default tree structure. In the above Main syntax, the project indicates to write out an entity called RouterRules that contains all the statements that are parsed. Also note the use the valuesof keyword above which pulls the list of values up a level allowing an extra set of braces, that would be generated by default, to be removed.

Next the syntax for Statement is defined which will consist of and if token and a FilterType syntax. The Statement syntax statement looks like this (line 5):

syntax Statement = TkIf a:FilterType => {Filter => a};

The a variable is used to store the value of a FilterType which is written out to an Entity called Filter.

Line 33 is one of the more interesting parts of the grammar. This is where the syntax for parsing the URL list is defined. The grammar uses Rule Parameters to parse list of definitions. It works sort of like a method call, the Destination syntax takes in destination text and a separator and then splits up the list of URLs based on the separator, projecting each URL out as a value.

The listing below shows the complete “M” output of the example router configuration from Listing 1:

    1: module Router {
    2:     RouterRules {
    3:         {
    4:             Filter => {
    5:                 action => "header",
    6:                 field => "version",
    7:                 operator => "=",
    8:                 fieldvalue => "\"Precise\"",
    9:                 priority => "0",
   10:                 Destinations =>  {
   11:                     {
   12:                         URL => "https://localhost:9090/servicemodelsamples/regularcalc"
   13:                     }
   14:                 }
   15:             }
   16:         },
   17:         {
   18:             Filter => {
   19:                 action => "header",
   20:                 field => "version",
   21:                 operator => "=",
   22:                 fieldvalue => "\"Rounded\"",
   23:                 priority => "",
   24:                 Destinations =>  {
   25:                     {
   26:                         URL => "https://localhost:9091/servicemodelsamples/roundingcalc"
   27:                     }
   28:                 }
   29:             }
   30:         },
   31:         {
   32:             Filter => {
   33:                 action => "endpoint",
   34:                 operator => "=",
   35:                 field => "",
   36:                 fieldvalue => "\"ServiceA\"",
   37:                 priority => "",
   38:                 Destinations =>  {
   39:                     {
   40:                         URL => "https://localhost:9090/servicemodelsamples/regularcalc"
   41:                     },
   42:                     {
   43:                         URL => "https://localhost:9091/servicemodelsamples/roundingcalc"
   44:                     }
   45:                 }
   46:             }
   47:         }
   48:     }
   49: }

Listing 3

The Router Model

In order to process the “M” instance data generated from our language using the “M” tool chain the above a model is needed. The instance data output is too ambiguous for “M” to know how to store it in the Repository database. The “M” language is used to define a schema for the router configurations model. A schema (also referred to as model) in “M” is a definition of a data type, that is, the metadata that describes a type’s mandatory and optional attributes. In database terminology, you can think of a schema as similar to the definition of a table.

In the RouterManager sample the model definition is stored in a file called RouterModel.m. The model is defined as being in the Router module which is the same as the RouterManager grammar and the router configuration instance data shown above.

The router configuration model will have an entity called RouterRule which will have a unique id and a reference to a router filter that makes up the rule. The definition of RouterRule is shown below:

    1: type RouterRule
    2: {
    3:     Filter : FilterType? where value in FilterTypes;
    4:     id : Integer32 => AutoNumber();    
    5: } where identity(id);

Listing 4

Entities are defined using the type keyword. A FilterType entity is defined to for an action which is a text field that holds the name of the type of filter used in the rule. It also contains an optional field that indicated what value should be inspected by a header rule. The operator type is used to specify if the value should be compared as equal or one of the other comparisons (=, contains, startswith, endswith). The FilterType entity has a field called fieldvalue, which contains the value that needs to be matched by the incoming message. The priority field is used to change the order the filter is evaluated. Lastly the FilterType entity has a unique identifier as well as a Destinations field that contains a reference to one or more destination entities.

The listing for FilterType is shown below:

    1: type FilterType
    2: {
    3:     id : Integer32 => AutoNumber();    
    4:     action: Text(20);
    5:     field: Text?;
    6:     operator : Text;
    7:     fieldvalue: Text;
    8:     priority: Text;
    9:     Destinations : {(DestinationType where value in DestinationList)*};
   10: } where identity(id);

Listing 5

The DestinationType entity is used to store a URL of an endpoint to send messages to. Each filter has one or more URLs that a message can be delivered to. The DestinationType entity has a unique id and a text field to store the URL. The Listing for DestinationType is shown below:

    1: type DestinationType 
    2: {
    3:     id : Integer32 => AutoNumber();
    4:     URL : Text;
    5: } where identity(id);

Listing 6

Note: The RouterManager grammar supports multiple operators, but the RouterManager runtime only supports the “=” operator. The other operators are left as an implementation exercise if the functionality is desired.

Type statements indicate what a model schema should look like but does not convey how it should be stored. Extents are used to specify how the entities are stored. The RouterModel defines three extents one for each of the types. DestinationList holds zero or more DestinationTypes, denoted by using the * operator. FilterTypes holds FilterType entities and RouterRules hold RouterRule entities

The complete listing of the RouterModel.m file is show below:

    1: module Router{
    2:     type DestinationType 
    3:     {
    4:         id : Integer32 => AutoNumber();
    5:         URL : Text;
    6:     } where identity(id);
    7:     
    8:     type FilterType
    9:     {
   10:         id : Integer32 => AutoNumber();    
   11:         action: Text(20);
   12:         field: Text?;
   13:         operator : Text;
   14:         fieldvalue: Text;
   15:         priority: Text;
   16:         Destinations : {(DestinationType where value in DestinationList)*};
   17:     } where identity(id);
   18:     
   19:     type RouterRule
   20:     {
   21:         Filter : FilterType? where value in FilterTypes;
   22:         id : Integer32 => AutoNumber();    
   23:     } where identity(id);
   24:     
   25:     DestinationList : {DestinationType*};
   26:     FilterTypes : {FilterType*};
   27:     RouterRules:{RouterRule*};
   28: }

Listing 4

Note: There is an easy way to see if your language output maps to your model. In “Intellipad” load up your “M” model definition and paste in some “M” instance data generated from the language input text (see Listing 3). “Intellipad” will show error where the instance data doesn’t match the model. You can then change the model to better match the instance data or figure out how to change your projections in the grammar to match the model. In the RouterManager sample, if you have run the redeploy_router.bat file successfully, you should see a file called Router.m that contains the sample configuration instance data. Copy one of the RouterRules entity data to the clipboard and then load the RouterModel.m file into “Intellipad”. Insert the copied RouterRules entity from the clipboard below the RouteRules extent definition. In the “Intellipad” “M Mode” menu, choose “T-SQL Preview”. It should show the T-SQL in a second window with no errors as the grammar output matches the model. Image 2 below is a screen shot of the RouterModel with a RouterRules entity data inserted in “Intellipad” (notice the RouterRules data highlighted at the bottom):

Image 2

Conclusion

As you can see, it is very easy to define domain-specific languages and data models using the SQL Server Modeling CTP. By combining the two, it gives you the foundation to make powerful model driven applications. Part III of the “Model Driven Content Based Routing using the SQL Server Modeling CTP” will cover the RouterManager runtime and how it uses the model and model data for message routing.