Complex Event Processing (CEP) solution with Business Rule Engine

I have implemented CEP solution with BizTalk and focused on real time passenger flow through the airport scenario. Imagine that information of passengers are flowing through the pipeline and you want to define some criteria to filter passenger data over pipeline. In CEP concept, collection of criteria or rules is assumed as a event. You may want to collect many different events data from multiple sources simultaneously. As a result, the aim of the complex event processing is to identify meaningful opportunities, threads etc.

solution
Let’s take a look at CEP implementation. The solution has two orchestration which Detects event and process event orchestration. The detect event orchestration takes passenger info and checks whether any event occurs or not. Every detected event is stored to event table in database. Also the event which reaches the max occurrence criterion is published to BizTalk MessageBox. The second orchestration catches the published event(thread or opportunity) and tries to process.

But where is the Business rule engine in this solution? Business rule engine has responsibility to execute event rules in detect orchestration. Can we define new event(s) without deploying the solution again? Yes, the events can be defined using with Business Rule Composer.

First, we need to define our schema to represent passenger and event information.

Schema_PassengerInfo Schema_Event

Then I have created two events that are called Turkish and German Passenger Events with Business Rule Composer.
Event_German

Event_Turkish

Business Rule Engine takes passenger info and executes defined rules of event over passenger info. Then If passenger info complaint with rules, BRE sets the values that are event name, max occurrence of event and description. This is also tricky point BRE determines events dynamically depends on the passenger info.

After defining rules, we can construct our orchestration. Passenger info comes from Port_PI receive port. And the event info is initialized in expression shape.

Orch1

We send passenger and event info to Business Rule Engine. Instead of using call rules shape in orchestration, there is a rule executer helper in the solution.
Orch2

namespace BT.POC.RuleExecuter
{
    [Serializable]
    public class PolicyUtility
    {
        public string Execute(XLANGMessage msgPassenger, XLANGMessage msgEvent)
        {
            string output = String.Empty;
            XmlDocument xmlDocPassenger = (XmlDocument)msgPassenger[0].RetrieveAs(typeof(XmlDocument));
            XmlDocument xmlDocEvent = (XmlDocument)msgEvent[0].RetrieveAs(typeof(XmlDocument));

            TypedXmlDocument txdPassenger = new TypedXmlDocument("ibrahimoguz.BiztalkPOC.CEP.Schemas.PassengerInfo", xmlDocPassenger);
            TypedXmlDocument txdEvent = new TypedXmlDocument("ibrahimoguz.BiztalkPOC.CEP.Schemas.EventSchema", xmlDocEvent);

            Microsoft.BizTalk.RuleEngineExtensions.RuleSetDeploymentDriver breDriver = new Microsoft.BizTalk.RuleEngineExtensions.RuleSetDeploymentDriver();
            Microsoft.RuleEngine.RuleStore breStore = breDriver.GetRuleStore();

            Microsoft.RuleEngine.RuleSetInfoCollection colPolInfo = null;
            colPolInfo = breStore.GetRuleSets(RuleStore.Filter.All);

            string nameXPath = "/*[local-name()='EventInfo' and namespace-uri()='http://ibrahimoguz.BiztalkPOC.CEP.EventSchema']/*[local-name()='Name' and namespace-uri()='']";
            foreach (RuleSetInfo pInfo in colPolInfo)
            {
                Policy p = new Policy(pInfo.Name);
                object[] facts = new object[2];
                facts[0] = txdPassenger;
                facts[1] = txdEvent;
                p.Execute(facts);

                XmlNode nameNode = txdEvent.Document.SelectSingleNode(nameXPath);
                if (nameNode != null && !String.IsNullOrWhiteSpace(nameNode.InnerText))
                {
                    output = txdEvent.Document.OuterXml;
                    break;
                }
            }

            return output;
        }
    }
}

The PolicyUtility class has execute method that takes two orchestration passenger and event messages. This utility fetches all policies from business rule engine rules store and executes sequentially. If any event occurrence returns event information to orchestration.

eventIf

If the event is occurred then the number of occurrence of this event checked from database.
eventcheck

public int GetCountEventOccurrence(string eventName)
{
   SqlConnection con = new SqlConnection() { ConnectionString = conString };
   SqlCommand command = new SqlCommand()
   {
      Connection = con,
      CommandType = System.Data.CommandType.Text,
      CommandText = String.Format("SELECT COUNT(*) FROM Event WHERE EventName='{0}' AND IsProcessed = 0", eventName)
    };

    con.Open();
    int result = (int)command.ExecuteScalar();
    con.Dispose();
    return result;
}

The passenger info mapped to event into before persisting event info to database.
Map

persist

maxOccurrence

OrchDetectLast

When max occurrence of event is reached then isProcessed field of event is set to true.
setEventProcessed

Then message is published into messageBox to trigger EventProcessing work flow.
EventProcessing

This is just testing purpose therefore event processing orchestration not have business logic. Let’s make some tests. I have used following passenger message. I have sent same message to reach max event occurrence number.

<ns0:PassengerInfo>
  <Passenger>
   <Name>Name_0</Name>
   <Surname>Surname_0</Surname>
   <FlightNo>FlightNo_0</FlightNo>
   <Departure>Frankfurt Airport</Departure>
   <Arrival>Vienna Airport</Arrival>
   <Date>2015-03-14</Date>
   <Airline>Lufthansa Airlines</Airline>
   <ScheduledTime>1999-05-31T13:20:00.000-05:00</ScheduledTime>
   <EstimatedTime>1999-05-31T13:20:00.000-05:00</EstimatedTime>
   <Age>10</Age>
   <Genre>F</Genre>
   <Nationality>DEUTCH</Nationality>
  </Passenger>
</ns0:PassengerInfo>

Eventlogs are as follows;
t1

t2

t3

t4

t45PNG

t6

t7

t8

Advertisements

Mapping BizTalk Xsd Types to .NET Types

The following table shows how Wcf svcUtils.exe transforms the schema data types to .NET types. The detailed explanation can be found in the SOA Patterns with BizTalk Server 2009 book of Ricard Seroter.

BizTalk XSD Type Description .NET Type
anyURI Can be any absolute or relative Uniform Resource Identifier Reference System.String
base64Binary Holds Base64-encoded arbitrary binary data System.Byte[]
boolean Logical value(0/1 or true/false) System.Boolean
byte Holds an 8-bit value System.SByte
date Object with year, month, and day properties System.DateTime
dateTime Object with year, month, day, hour, minute, second, and timezone properties System.DateTime
decimal Contains a subset of real numbers with support for at a minimum of 18 decimal digits System.Decimal
double A double precision 64-bit floating point type System.Double
duration Represents a duration of time consisting of year, month, day, hour, minute and second System.String
ENTITIES Separated list of ENTITY references System.String
ENTITY Unparsed entity that may include non-XML content System.String
float A single precision 32-bit floating point type System.Single
gDay Equal to a day recurring each month System.String
gMonth Equal to a month recurring each year System.String
gMonthDay A calendar date (month + day) recurring each year System.String
gYear A period of a single year System.String
gYearMonth A particular calendar month in a specific year System.String
hexBinary Represents arbitrary hex-encoded binary data System.Byte[]
ID Definition of document-global unique identifiers System.String
IDREF A reference to a unique identifier (ID) System.String
IDREFS Separated list of IDREF references System.String
int A 32-bit signed integer System.Int32
integer A signed integer of arbitrary length System.String
language Set of language codes called out in RFC 3066 (e.g. en-US) System.String
long 64-bit signed integer System.Int64
Name XML string with no whitespace System.String
NCName Name that conforms to namespace standard (e.g. no colons) System.String
negativeInteger Encompasses all strictly negative integers System.String
NMTOKEN Set of XML “name tokens” excluding spaces or commas System.String
NMTOKENS Separated list of NMTOKENS System.String nonNegativeInteger Encompasses all positive integers (including zero) System.String
nonPositiveInteger Encompasses all negative integers (including 0) System.String
normalizedString Contains whitespace-replaced strings (meaning all carriage returns, tabs, etc have been replaced) System.String
positiveInteger Encompasses all strictly positive integers System.String
QName Qualified name as a combination of namespace name and part name XmlQualifiedName
short Set of 16-bit integers System.Int16

BizTalk Mapping – Inline XSLT usage

The BizTalk Mapper has many functoids that are capable of transforming operations. There are different type of functoids such as Advanced functoids(Value mapping, Table Looping, Assert etc.), String functoids.

2

 

In some cases, the functoids that are listed in toolbox are not capable of solving our transformation issues such as sorting, grouping etc. In order to solve complex and advance transformation issues we have to use inline xslt using with scripting functoid. It is shown in figure below.

1

 

I have tried to list most used xslt commands and i have used following source xml in the examples.


SOURCE XML
----------
<ns0:Root xmlns:ns0="http://schemas.microsoft.com/BizTalk/2003/aggschema">
 <InputMessagePart_0>
  <ns1:OrderDoc xmlns:ns1="http://MessageEnrichment.Schema.OrderDoc">
   <Id>10</Id>
   <CustomerName>CustomerName_0</CustomerName>
  </ns1:OrderDoc>
 </InputMessagePart_0>
 <InputMessagePart_1>
  <ns2:GetOrderDetailsResponse xmlns:ns2="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo">
   <ns2:StoredProcedureResultSet0>
    <ns3:StoredProcedureResultSet0 xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
     <ns3:OrderId>10</ns3:OrderId>
     <ns3:OrderItemId>10</ns3:OrderItemId>
     <ns3:ProductId>10</ns3:ProductId>
     <ns3:Quantity>10</ns3:Quantity>
     <ns3:Name>10_Name</ns3:Name>
     <ns3:Description>10_Description</ns3:Description>
     <ns3:Price>10.4</ns3:Price>
    </ns3:StoredProcedureResultSet0>
    <ns3:StoredProcedureResultSet0 xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
     <ns3:OrderId>11</ns3:OrderId>
     <ns3:OrderItemId>11</ns3:OrderItemId>
     <ns3:ProductId>11</ns3:ProductId>
     <ns3:Quantity>11</ns3:Quantity>
     <ns3:Name>11_Name</ns3:Name>
     <ns3:Description>11_Description</ns3:Description>
     <ns3:Price>11.4</ns3:Price>
    </ns3:StoredProcedureResultSet0>
    <ns3:StoredProcedureResultSet0 xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
     <ns3:OrderId>12</ns3:OrderId>
     <ns3:OrderItemId>12</ns3:OrderItemId>
     <ns3:ProductId>12</ns3:ProductId>
     <ns3:Quantity>12</ns3:Quantity>
     <ns3:Name>12_Name</ns3:Name>
    <ns3:Description>12_Description</ns3:Description>
    <ns3:Price>12.4</ns3:Price>
    </ns3:StoredProcedureResultSet0>
  </ns2:StoredProcedureResultSet0>
  <ns2:ReturnValue>15</ns2:ReturnValue>
  </ns2:GetOrderDetailsResponse>
 </InputMessagePart_1>
</ns0:Root>

 

**********************************************************************
                      ASSIGN SELECTED TO VARIABLE
**********************************************************************

XSLT
------
<xsl:variable name="vOrderName"
              xmlns:ns2="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo"
              xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails"
              select="//InputMessagePart_1/ns2:GetOrderDetailsResponse/ns2:StoredProcedureResultSet0/ns3:StoredProcedureResultSet0" />

<OrderName>
  <xsl:for-each select="$vOrderName" xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
    <Name>
      <xsl:element name="Description"><xsl:value-of select="ns3:Description"/></xsl:element>
      <xsl:element name="Quantity"><xsl:value-of select="ns3:Quantity"/></xsl:element>
      <xsl:element name="Price"><xsl:value-of select="ns3:Price"/></xsl:element>
      <xsl:element name="OrderItemId"><xsl:value-of select="ns3:OrderItemId"/></xsl:element>
    </Name>
  </xsl:for-each>
</OrderName>

&nbsp;

TARGET XML
----------
<OrderName>
  <Name>
    <Description>10_Description</Description>
    <Quantity>10</Quantity>
    <Price>10.4</Price>
    <OrderItemId>10</OrderItemId>
  </Name>
  <Name>
    <Description>11_Description</Description>
    <Quantity>11</Quantity>
    <Price>11.4</Price>
    <OrderItemId>11</OrderItemId>
  </Name>
  <Name>
    <Description>12_Description</Description>
    <Quantity>12</Quantity>
    <Price>12.4</Price>
    <OrderItemId>12</OrderItemId>
  </Name>
</OrderName>

 

**********************************************************************
                                 IF
**********************************************************************

XSLT
------
<xsl:variable name="vOrderName"
              xmlns:ns2="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo"
              xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails"
              select="//InputMessagePart_1/ns2:GetOrderDetailsResponse/ns2:StoredProcedureResultSet0" />

<OrderName>
  <xsl:for-each select="$vOrderName" xmlns:ns2="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo" xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
  <StoredProcedureResultSet0_ns2>
    <xsl:if test="count(ns2:StoredProcedureResultSet0)>0">
      Node sayisi <xsl:value-of select="count(ns2:StoredProcedureResultSet0)"/>
    </xsl:if>
  </StoredProcedureResultSet0_ns2>
  <StoredProcedureResultSet0_ns3>
    <xsl:if test="count(ns3:StoredProcedureResultSet0)>0">
      Node sayisi <xsl:value-of select="count(ns3:StoredProcedureResultSet0)"/>
    </xsl:if>
  </StoredProcedureResultSet0_ns3>
 </xsl:for-each>
</OrderName>

&nbsp;

TARGET XML
----------
<OrderName>
  <StoredProcedureResultSet0_ns2 />
  <StoredProcedureResultSet0_ns3>
    Node sayisi 3
  </StoredProcedureResultSet0_ns3>
</OrderName>

 

**********************************************************************
                            CHOOSE, WHEN
**********************************************************************

XSLT
------
<xsl:variable name="vOrderName"
              xmlns:ns2="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo"
              xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails"
              select="//InputMessagePart_1/ns2:GetOrderDetailsResponse/ns2:StoredProcedureResultSet0/ns3:StoredProcedureResultSet0" />

<OrderName>
  <xsl:for-each select="$vOrderName" xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
    <xsl:variable name="quantityValue" select="ns3:Quantity" />
    <xsl:if test="$quantityValue=10">
      <xsl:element name="Quantity">If case - <xsl:value-of select="ns3:Quantity"/></xsl:element>
    </xsl:if>
    <xsl:choose>
      <xsl:when test="$quantityValue>10">
        <xsl:element name="Quantity"><xsl:value-of select="ns3:Quantity"/></xsl:element>
      </xsl:when>
      <xsl:otherwise>
        <xsl:element name="Quantity">Not enough - <xsl:value-of select="$quantityValue" /></xsl:element>
      </xsl:otherwise>
    </xsl:choose>
  </xsl:for-each>
</OrderName>

&nbsp;

TARGET XML
----------
<OrderName>
  <Quantity>If case - 10</Quantity>
  <Quantity>Not enough - 10</Quantity>
  <Quantity>11</Quantity>
  <Quantity>12</Quantity>
</OrderName>

 

**********************************************************************
                                SORTING
**********************************************************************

XSLT
------
<xsl:variable name="vOrderName"
              xmlns:ns2="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo"
              xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails"
              select="//InputMessagePart_1/ns2:GetOrderDetailsResponse/ns2:StoredProcedureResultSet0/ns3:StoredProcedureResultSet0" />

<OrderName>
  <xsl:for-each select="$vOrderName" xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
    <xsl:sort select="ns3:Description" order="descending"/>
    <xsl:element name="Description"><xsl:value-of select="ns3:Description"/></xsl:element>
  </xsl:for-each>
</OrderName>

TARGET XML
----------
<OrderName>
  <Description>12_Description</Description>
  <Description>11_Description</Description>
  <Description>10_Description</Description>
</OrderName>

XSLT
------
<xsl:variable name="vOrderName"
              xmlns:ns2="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo"
              xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails"
              select="//InputMessagePart_1/ns2:GetOrderDetailsResponse/ns2:StoredProcedureResultSet0/ns3:StoredProcedureResultSet0" />

<OrderName>
  <xsl:for-each select="$vOrderName" xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
    <xsl:sort select="ns3:Quantity" order="descending" data-type="number"/>
    <xsl:element name="Quantity"><xsl:value-of select="ns3:Quantity"/></xsl:element>
  </xsl:for-each>
</OrderName>

TARGET XML
----------
<OrderName>
  <Quantity>12</Quantity>
  <Quantity>11</Quantity>
  <Quantity>10</Quantity>
</OrderName>

 


**********************************************************************
                             GROUPING
**********************************************************************

SOURCE XML
-----------
<ns0:Root xmlns:ns0="http://schemas.microsoft.com/BizTalk/2003/aggschema">
<InputMessagePart_0>
<ns1:OrderDoc xmlns:ns1="http://MessageEnrichment.Schema.OrderDoc">
<Id>10</Id>
<CustomerName>CustomerName_0</CustomerName>
</ns1:OrderDoc>
</InputMessagePart_0>
<InputMessagePart_1>
<ns2:GetOrderDetailsResponse xmlns:ns2="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo">
<ns2:StoredProcedureResultSet0>
<ns3:StoredProcedureResultSet0 xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
<ns3:OrderId>10</ns3:OrderId>
<ns3:OrderItemId>10</ns3:OrderItemId>
<ns3:ProductId>10</ns3:ProductId>
<ns3:Quantity>10</ns3:Quantity>
<ns3:Name>10_Name</ns3:Name>
<ns3:Description>10_Description</ns3:Description>
<ns3:Price>10.4</ns3:Price>
</ns3:StoredProcedureResultSet0>
<ns3:StoredProcedureResultSet0 xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
<ns3:OrderId>10</ns3:OrderId>
<ns3:OrderItemId>10</ns3:OrderItemId>
<ns3:ProductId>10</ns3:ProductId>
<ns3:Quantity>10</ns3:Quantity>
<ns3:Name>10_Name</ns3:Name>
<ns3:Description>10_Description</ns3:Description>
<ns3:Price>10.4</ns3:Price>
</ns3:StoredProcedureResultSet0>
<ns3:StoredProcedureResultSet0 xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
<ns3:OrderId>11</ns3:OrderId>
<ns3:OrderItemId>11</ns3:OrderItemId>
<ns3:ProductId>11</ns3:ProductId>
<ns3:Quantity>11</ns3:Quantity>
<ns3:Name>11_Name</ns3:Name>
<ns3:Description>11_Description</ns3:Description>
<ns3:Price>11.4</ns3:Price>
</ns3:StoredProcedureResultSet0>
<ns3:StoredProcedureResultSet0 xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
<ns3:OrderId>11</ns3:OrderId>
<ns3:OrderItemId>11</ns3:OrderItemId>
<ns3:ProductId>11</ns3:ProductId>
<ns3:Quantity>11</ns3:Quantity>
<ns3:Name>11_Name</ns3:Name>
<ns3:Description>11_Description</ns3:Description>
<ns3:Price>11.4</ns3:Price>
</ns3:StoredProcedureResultSet0>
<ns3:StoredProcedureResultSet0 xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
<ns3:OrderId>12</ns3:OrderId>
<ns3:OrderItemId>12</ns3:OrderItemId>
<ns3:ProductId>12</ns3:ProductId>
<ns3:Quantity>12</ns3:Quantity>
<ns3:Name>12_Name</ns3:Name>
<ns3:Description>12_Description</ns3:Description>
<ns3:Price>12.4</ns3:Price>
</ns3:StoredProcedureResultSet0>
<ns3:StoredProcedureResultSet0 xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails">
<ns3:OrderId>13</ns3:OrderId>
<ns3:OrderItemId>13</ns3:OrderItemId>
<ns3:ProductId>13</ns3:ProductId>
<ns3:Quantity>13</ns3:Quantity>
<ns3:Name>13_Name</ns3:Name>
<ns3:Description>13_Description</ns3:Description>
<ns3:Price>13.4</ns3:Price>
</ns3:StoredProcedureResultSet0>
</ns2:StoredProcedureResultSet0>
<ns2:ReturnValue>15</ns2:ReturnValue>
</ns2:GetOrderDetailsResponse>
</InputMessagePart_1>
</ns0:Root>

XSLT
----
<xsl:variable name="vStoredProcedureResultSet0"
              xmlns:ns2="http://schemas.microsoft.com/Sql/2008/05/TypedProcedures/dbo"
              xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails"
              select="//InputMessagePart_1/ns2:GetOrderDetailsResponse/ns2:StoredProcedureResultSet0/ns3:StoredProcedureResultSet0" />

<xsl:variable name="vQuantity"
              xmlns:ns3="http://schemas.microsoft.com/Sql/2008/05/ProceduresResultSets/dbo/GetOrderDetails"
              select="//ns3:StoredProcedureResultSet0[not(ns3:Quantity=preceding-sibling::ns3:StoredProcedureResultSet0/ns3:Quantity)]/ns3:Quantity" />

<OrderName>
  <xsl:for-each select="$vQuantity">
    <Quantity><xsl:value-of select="."/></Quantity>
  </xsl:for-each>
</OrderName>

TARGET XML
----------
<OrderName>
  <Quantity>10</Quantity>
  <Quantity>11</Quantity>
  <Quantity>12</Quantity>
  <Quantity>13</Quantity>
</OrderName>

 

**********************************************************************
                              PIVOT
**********************************************************************

SOURCE XML
----------

<Data Header="AAA" date="2008-10-28" Name="a1" Value="1.0" />
<Data Header="AAA" date="2008-10-28" Name="a2" Value="2.0" />
<Data Header="AAA" date="2008-10-28" Name="a3" Value="3.0" />
<Data Header="BBB" date="2008-10-28" Name="a1" Value="1.0" />
<Data Header="BBB" date="2008-10-28" Name="a2" Value="2.0" />
<Data Header="BBB" date="2008-10-28" Name="a3" Value="3.0" />

XSLT
------
<xsl:template name="NameValueTemplate">
  <xsl:param name="param1" />
  <xsl:for-each select="//Data[@Header=$param1]">
    <xsl:element name="Name"><xsl:value-of select="@Name" /></xsl:element>
    <xsl:element name="Value"><xsl:value-of select="@Value" /></xsl:element>
  </xsl:for-each>
</xsl:template>

<xsl:element name="Data">
  <xsl:for-each select="Data[not(@Header=preceding-sibling::Data/@Header)]">
    <xsl:element name="Header"><xsl:value-of select="@Header" /></xsl:element>
    <xsl:element name="date"><xsl:value-of select="@date" /></xsl:element>
    <xsl:element name="Record">
      <xsl:call-template name="NameValueTemplate">
        <xsl:with-param name="param1" select="string(@Header)" />
      </xsl:call-template>
    </xsl:element>
  </xsl:for-each>
</xsl:element>

&nbsp;

TARGET XML
----------
<Data>
  <Header>AAA</Header>
  <date>2008-10-28</date>
  <Record>
    <Name>a1</Name>
    <Value>1.0</Value>
    <Name>a2</Name>
    <Value>2.0</Value>
    <Name>a3</Name>
    <Value>3.0</Value>
  </Record>
  <Header>BBB</Header>
  <date>2008-10-28</date>
  <Record>
    <Name>a1</Name>
    <Value>1.0</Value>
    <Name>a2</Name>  
    <Value>2.0</Value>
    <Name>a3</Name>
    <Value>3.0</Value>
  </Record>
</Data>

 

BizTalk Server 2013 Hot fixes

While configuring BAM section, BizTalk configuration tool completes the action with error that is detailed in log file. The error is classic access problem to SQL database. “A network-related or instance-specific error occurred” is irrelevant because the users that will do operation in sql database are defined on the database. I have searched the problem through the internet and have found the solution which is all related with a bug. Microsoft has launched the hot fixes in order to solve that kind of problems.

 

To apply this cumulative update package, you must have BizTalk Server 2013 installed.

Cumulative update package 1 for BizTalk Server 2013 – here

  • FIX: User cannot perform certain database-related operations in BizTalk Server 2013 
  • FIX: The vertical scroll bar on the target schema does not work correctly when you use Visual Studio to design a BizTalk Server 2013 map
  • FIX: BAM tools cannot be configured in a multi-node BizTalk Server 2013 environment

 

Cumulative update package 2 for BizTalk Server 2013 – here

  • FIX: The value for the “maxOccurs” attribute is invalid when you validate the BizTalk X12 EDI 835 schema in a BizTalk EDI application in BizTalk Server 2010 or in in BizTalk Server 2013
  • FIX: The memory that a BizTalk host process consumes increases when there are no messages in the queue in BizTalk Server 2010 or in BizTalk Server 2013
  • FIX: “The InnerText property is write only” error message when you use System.Xml.XmlDocument in BizTalk Server 2010 or in BizTalk Server 2013
  • FIX: Pipeline on receive locations reverts to pass-through after you update the schema assembly in BizTalk Server 2010 or in BizTalk Server 2013
  • FIX: “Could not load file or assembly ‘Microsoft.BizTalk.Interop.SSOClient Version=5.0.1.0” error when you use the ESB configuration tool in BizTalk Server 2013
  • FIX: SQL connection leak when you use WCF-based SQL Adapter one-way send port to return a result set from a database in BizTalk Server 2010 or in BizTalk Server 2013
  • FIX: 2-way WCF-BasicHttp receive port does not respond to a WCF client when the “failed message routing” feature is enabled in BizTalk Server 2010 or in BizTalk Server 2013
  • FIX: Message is suspended when you try to receive an encrypted AS2 message together with asynchronous MDN in BizTalk Server 2010 or in BizTalk Server 2013
  • FIX: An update for Message Box Viewer is available in Microsoft BizTalk Server 2013 cumulative update 2
  • FIX: Support for HIX EDI transactions 005010×306 (820) and 005010×220 (834) for BizTalk Server 2013, BizTalk Server 2010, and BizTalk Server 2009
  • FIX: XpathMutatorStream class does not work as expected when you execute Xpath statements against XmlDocuments in BizTalk Server 2013
  • FIX: BizTalk Server 2013 cannot receive the messages from the view in document library or from the SharePoint online library
  • FIX: High CPU usage after you install BizTalk Server 2013 and SQL Server 2012 on a computer
  • FIX: “System.ArgumentException” error when you browse WCF services from ESB Toolkit 2.2 in BizTalk Server 2013

BizTalk high performance tracing/logging – CAT, ETW

Every BizTalk developer needs and uses the tracing and logging feature especially development environment. Some circumstances we need to log and trace something in production environment even though this negatively impacts the performance of production environment. Basically System.Diagnostic.Trace in the .NET Framework uses for tracing but it is so primitive and doesn’t give us the flexibility of start and stop trace capability. Additionally  log4net and Enterprise library are flexible and have more features. This great article compares the performance of  tracing components which are System.Diagnostic.Trace, Entireprise library tracing, log4net and Event Tracing for Windows.

Benchmarked Components Performance Results

ETW has the most performance and it is 25 times greater than Log4net Tracing Component. The performance of System.Diagnostic component is the least one and it shouldn’t be used in production environment if the performance is required.

The BizTalk CAT Instrumentation Framework is a high performance tracing/logging framework for BizTalk that builds upon the Event Tracing for Windows (ETW) infrastructure. The BizTalk CAT Instrumentation Framework is available here. The CAT Tracing component can be started or stopped using with BizTalk CAT Instrumentation Framework Controller.

8

I have developed the POC application for examining features of BizTalk CAT Tracing Component.

4

3

9

Deployment Framework for BizTalk (BTDF)

Overview

The deployment of BizTalk applications can be error-prone if you don’t organize BizTalk artifacts properly. Deployment Framework for BizTalk (BTDF) is very useful toolkit, fastest way of deploying BizTalk applications and automates the entire BizTalk application deployment and update processes. BTDF can be found in biztalkdeployment.codeplex.com also BTDF technical guideline is here. The release 5.1 is compatible with BizTalk2013 and Visual Studio 2013. While setting up release 5.0, the deployment project never seen in visual studio create project dialog.

The Architecture of BTDF

Architecture of BTDF

Architecture of BTDF

Steps of using BTDF

After installing BTDF, Deployment Framework for BizTalk project template can be selected to create into your solution which is going to be deployed.

1

2

3

4

btdfproj fils important because all configuration can be done about deployment. For example you can set your BizTalk host name, also define your third party or custom components in here. The best practice can be said that Orchestrations, schemas and maps shall be seperated different projects or dlls rather than collecting same project. In my POC My orchestration, schemas and transforms are in same project and it’s output is SQLAdapterPOC.dll.


<!-- !!! TODO !!! -->

<!-- Add ItemGroup elements that contain one or more Schemas, Orchestrations, Transforms, etc. elements that describe -->
 <!-- the specific artifacts in your solution that need to be deployed. Use IntelliSense as a guide. -->
 <ItemGroup>
 <BizTalkHosts Include="BizTalkServerApplication"/>
 <Components Include="SQLAdapterPOC.dll">
 <LocationPath>..\$(ProjectName)\bin\$(Configuration)</LocationPath>
 </Components>
 <Orchestrations Include="SQLAdapterPOC.dll">
 <LocationPath>..\$(ProjectName)\bin\$(Configuration)</LocationPath>
 </Orchestrations>
 <Schemas Include="SQLAdapterPOC.dll">
 <LocationPath>..\$(ProjectName)\bin\$(Configuration)</LocationPath>
 </Schemas>
 <Transforms Include="SQLAdapterPOC.dll">
 <LocationPath>..\$(ProjectName)\bin\$(Configuration)</LocationPath>
 </Transforms>
 </ItemGroup>
 

5

Click Build Server Deploy MSI. If the configuration file is well organized, the msi is generated successfully.
6

In configuration file, you mention that includeOrchestration is true, you should specify the assembly file in the item group. Otherwise BTDF will give error which is specified (or default) file can not be loaded.

7

9

10

11

12

13

You need to delete the application is already deployed using with BizTalk Administration console. MSBuild.exe gives error SQLAdapter application is already exist.

14

15

The BizTalk project assembly and artifacts are deployed properly. Deploy result file can be seen below is created by BTDF. The components are deployed to GAC. BizTalk host service and IIS is restarted. The deployment finished automated way.

16

Publish Orchestration as WCF Service

Depending on our SOA solution architectuıre,   sometimes we have decided to communicate with clients using with http or https protocol. In order to client is able to send requests to BizTalk orchestration, we need to expose BizTalk Orchestrations as service (wcf or .asmx) in some SOA scenarios. BizTalk WCF Service Publishing Wizard can help us on this issue.

There is one important point which the assembly of Biztalk project should be placed in GAC. You can find the steps of wizard as follows:

1

2

3

 

“Add a Service Bus endpoint” should be selected if you want to deploy Biztalk project to Cloud.

 

 

4

5

 

BizTalk assembly should also be placed in GAC in order to process BizTalk orchestration correctly.

6

7

8

9

10

12

13

14

15

16