.Net Liberator

Windows Communication Foundation (WCF) is Microsoft’s unified programming model for building service-oriented applications. It enables developers to build secure, reliable, transacted solutions that integrate across platforms and interoperate with existing investments.

In order to communicate with a WCF service you will need a client proxy which can be generated automatically using Visual Studio by “Add Service Reference” but I don’t prefer this way because it generates too much code, extra configuration and the service must publish its metadata, besides that every time the service updated you will need to update the service reference in order to reflect the new changes.

To perfectly loose-couple your application layers you will need to separate service contracts into their own assemblies so they can be shared and consumed by different applications and layers but this approach leaves a lot of manual work for the clients consuming our services as each client has to implement proxy for the corresponding contract.

Through this talk I will show you a simple way to eliminate manual work during proxy development. I will use T4 to dynamically create proxy class for WCF service contract in the same assembly with the contract in order to simplify consumption of our WCF service. Finally in the talk I will show you how to create Visual Studio template to generate WCF proxy classes.

Before digging into WCF proxy generation, let’s have a look to how T4 works. T4 text template is a mixture of text blocks and control logic that can generate a text file. The control logic is written as fragments of program code in Visual C# or Visual Basic. The generated file can be text of any kind, such as a Web page, or a resource file, or program source code in any language.

The first thing we would do is to add T4 file to our solution, To do this, in Solution Explorer, right-click your project and point to Add, and then click New Item. In the Add New Item dialog box select Text Template from the Templates pane. Name the file NewTemplate.tt. Notice that the Custom Tool property of the file is TextTemplatingFileGenerator.

Open the added file and change “hostspecific” to true, “debug” to true, “language” to C# and the output extension to “cs” like below:

<#@ template language="C#" hostspecific="True" debug="True" #>

This means that our generated file will be in C# code and we enabled “hostspecific” in order to allow our T4 to access the text template host engine. Make sure that “System.ServiceModel” DLL is referenced. In T4 add the below directives:

<#@ Assembly Name="EnvDTE" #>

<#@ Assembly Name="EnvDTE80" #>

<#@ import namespace="EnvDTE" #>

<#@ Import Namespace="EnvDTE80" #>

The above directives allow us to reference and access to visual studio environment along with text template generation engine. I have created a utility helper class (WCFProxyGenerator) that will provide the necessary helper functions to create proxy, these functions are summarized below:

1- RootNamespace: Gets the root namespace of the target project.
2- DefaultNamespace: Gets the default namespace for the target project.
3- GetServiceInterface(): Gets service interface type object for the specified file (searches the file for WCF interfaces).
4- GenerateParameters(): Gets a string that contains WCF method parameters.
5- GenerateCodeImports(): Gets the namespaces that need to be imported by the proxy.

Let’s now write some real T4 code, the first thing we need here is the name of the WCF service contract file, so I will declare T4 variable for it, also we can give our user the chance to choose the name of the proxy, if the user does not provide a certain name for the proxy then we will use the same name of the T4 file: 

    if (ServiceContract != null)
    {
        if (string.IsNullOrEmpty(ProxyName))
            ProxyName = System.IO.Path.GetFileNameWithoutExtension(this.Host.TemplateFile);
        WriteLine(Context.GenerateCodeImports(ServiceContract));
    }
    else
        return null;

#>

The above T4 code will create object of our utility class, then it will use that object to search for specified file in solution, if the contract is found then we will use utility object to import the required namespaces for our proxy “using statements”.

The next step is to write the T4 script that will generate WCF proxy class:

namespace <#=Context.DefaultNamespace #>

{      

       public partial class <#=ProxyName #> : ClientBase<<#=ServiceContract.Name #>>, <#=ServiceContract.Name #>

       {

       <#

       #>

       public <#=method.Type.AsString #> <#=method.Name #>(<#=Context.GenerateParameters(method.Parameters)#>)

              {

                     <#=method.Type.TypeKind == vsCMTypeRef.vsCMTypeRefVoid ? "" : "return" #> this.Channel.<#=method.Name #>(<#=Context.GenerateParameters(method.Parameters, false) #>);

              }

       <#

       }

}

In the above script, we first declare the namespace of our class then we generate partial class for the proxy that implements “ClientBase<>” along with our WCF service interface. I used a partial class in order to give other developer space for extending proxy implementation.

The T4 script then loops through each method in the WCF service contract and checks if the method is WCF operation method and if so it generates the method signature by using our utility class. After generating the method signature we fill the method body by using the channel factory that we have from our inherited base class.

I created a Visual Studio item template in order to simplify our proxy generation process. You can download the item template here (WCFProxyGenerator.zip (208.85 kb)) then drop this zip file as is in the appropriate location to be used by visual studio, the default location for this file should be “%USERPROFILE%\Documents\Visual Studio 2010.\Templates\ItemTemplates\”, restart visual studio and create a new class library project then add a new item to the project (you should see our item template there):

Add the WCF proxy generator template file, create any WCF service contract and open the T4 file and pass the service contract file name to T4 file then save the T4 file and you will get your WCF proxy class generated.

Microsoft Lync Server 2010 communications software offers instant messaging (IM), presence, conferencing, and telephony solutions that can support enterprise-level collaboration requirements. Recently I was working in an integration project that links Lync server with Microsoft exchange server and I found the best way to integrate these two entities is though EWS.

Exchange Web Services (EWS) provides the functionality to enable client applications to communicate with the Exchange server. EWS provides access to much of the same data that is made available through Microsoft Office Outlook. EWS clients can integrate Outlook data into Line-of-Business (LOB) applications. SOAP provides the messaging framework for messages sent between the client application and the Exchange server.

Though this talk I’m going to discuss EWS integration with .net client applications, in later articles I will discuss the integration with Lync Server 2010. To get started with EWS you will need to reference EWS to your solution, you can do this with the Visual Studio generated proxy classes for Exchange Web Services but I’m not going to use Microsoft managed APIs to interact with EWS that can be downloaded here.

The Microsoft Exchange Web Services (EWS) Managed API 1.1 provides a managed interface for developing client applications that use Exchange Web Services. The EWS Managed API simplifies the implementation of applications that communicate with Microsoft Exchange Server 2007 Service Pack 1 (SP1) and later versions of Microsoft Exchange. Built on the Exchange Web Services SOAP protocol and Autodiscover, the EWS Managed API provides a .NET interface to EWS that is easy to learn, use, and maintain.

The Exchange Web Services (EWS) Managed API consists of a single assembly, Microsoft.Exchange.WebServices.dll, from your Microsoft Visual Studio project, add a reference to Microsoft.Exchange.WebServices.dll, and add a using clause for the Microsoft.Exchange.WebServices.Data namespace.

The first thing we would do is to construct "ExchangeService" object that handles the communication with the exchange server:

The diagram below shows EWS object hierarchy:

Now we are ready to explore EWS features:

E-Mail

• Sending Email:

ExchangeService EWS = this.CreateExchangeService();
EmailMessage message = new EmailMessage(EWS);
message.Subject = "Hello EWS!";
message.Body = "Testing EWS.";
message.ToRecipients.Add("aalamir@intellecting.net");
message.SendAndSaveCopy();

• Replying Email:

In order to reply for an email, we should first search for the message we want to reply for as follows:

FindItemsResults<Item> Result = EWS.FindItems(WellKnownFolderName.Inbox,
   new SearchFilter.ContainsSubstring(ItemSchema.Subject, "Hello EWS!"),
   new ItemView(int.MaxValue) { PropertySet = new PropertySet(ItemSchema.Subject)});

   foreach (var itm in Result)
   {
       EmailMessage message = itm as EmailMessage;
       message.Reply("EWS Reply Test!!!", true);
   }

• Forwarding Email:

Meetings

Appointment meeting = new Appointment(EWS);

meeting.Subject = "EWS Meeting Test!!!";

meeting.Body = "Body Goes here.";

meeting.Start = DateTime.Now.AddDays(+1);

meeting.End = meeting.Start.AddMinutes(90);

meeting.Location = "Meeting Room[1]";

meeting.RequiredAttendees.Add("user1@domain.com");

meeting.RequiredAttendees.Add("user2@domain.com");

Now after exploring the main basic features of EWS, let’s dig through EWS notifications:

Notifications

EWS allows client applications to subscribe to event notifications that make it possible to determine what events occurred on a specific folder since a specific point in time (for example, what items were created, modified, moved, or deleted).

There are two types of subscriptions: pull subscriptions and push subscriptions. With pull subscription, the client application has to poll the server regularly to retrieve the list of events that occurred since the last time the server was polled. With push subscription, Exchange directly notifies the client application when an event occurs.

Using pull notifications with the EWS Managed API:

The following is an example that shows how to subscribe to pull notifications and how to retrieve the latest events.

Using push notifications with the EWS Managed API:

The EWS Managed API does not provide a built-in push notifications listener. It is the responsibility of the client application to implement such a listener.
I will not discuss the details of creating EWS notification listener as it is not the scope of this talk but I attached a full working solution that contains a simple WCF notification listener.

First,  decide which EWS items  you need to subscribe for, below we will subscribe for Email and calendar items:

Second, choose the event type you need, for example (New Mail, Deleted items, …):

List<EventType> eventTypes = new List<EventType>();
eventTypes.Add(EventType.NewMail);
eventTypes.Add(EventType.Deleted);
eventTypes.Add(EventType.Moved);
eventTypes.Add(EventType.Created);
eventTypes.Add(EventType.Modified);

The last thing is to subscribe for push notification:

PushSubscription pushSubscription = 
          service.SubscribeToPushNotifications(folderIds,
          new Uri(listenerEndpoint),1, null, eventTypes.ToArray());

you can find a full working solution of all the examples discussed above here EWS.zip (1.60 mb).

Pipelines are a component of Microsoft BizTalk Server that provides an implementation of the Pipes and Filters integration pattern. During the receiving and sending of messages, there are business reasons to perform transformations on messages to prepare them to enter or leave BizTalk Server. Pipelines enable the developer to define a series of transformations that will be performed on a message as it is being received or sent.

A common example is that you may need to compress your data before sending them to target subscriber in order to save server resources such as disk space or bandwidth, through this talk I'm going to discuss how to develop a general custom pipeline component to compress/decompress data. There are two types of pipelines, send and receive, and these match the ports in which they execute. Send pipelines are executed in send ports and in the response portion of a request/response receive port, while receive pipelines are executed in receive locations, and in the response portion of a solicit/response send port. Essentially, receive pipelines are intended to be used to transform messages that are being published to the MessageBox database, while send pipelines are intended to be used on messages which have been subscribed to and are being sent out of BizTalk Server.

Receive Pipeline Stages	Send Pipeline Stages
Decode: Decrypts or decodes the message data. Disassemble: Disassembles an interchange into smaller messages and parses message contents. Validate: Validates the message data, generally against a schema. Resolve Party: Identifies the BizTalk Server party associated with some security token in the message or message context.	Pre-assemble: Performs any message processing necessary before assembling the message. Assemble: Assembles the message and prepares it to be transmitted by taking steps such as adding envelopes, converting XML to flat files, or other tasks complementary to the disassemble stage in a receive pipeline. Encode: Encodes or encrypts the message before delivery.

A general pipeline component gets one message from the BizTalk Messaging Engine, processes it, and returns it to the BizTalk Server engine, it implements the following interfaces:

It has been a long time since my last talk about using service broker for notification of data changes, let's continue what we have started. The first thing we need to do here is how to receive a message from service broker. SQL server has a special command for retrieving a message from service broker:

WAITFOR ( RECEIVE TOP(@FetchSize) * FROM NotificationQueue), TIMEOUT @TimeToWait

As you observe from the above query that you can control the number of messages to fetch and the blocking timeout for a message to be received, if you have multiple messages submitted through one conversation handle then you can do batch receive but if one message is submitted then you can only receive one message at a time.
Now after we got our message from service broker, we want to prepare it to be audited. We supposed to transform the message body to XML format and use SQL server XML capabilities to transform the XML into SQL data.
I will create audit trace table with the following structure:

create table AuditTrace
(
   AuditTraceID bigint primary key identity(1,1),
   [User] nvarchar(200),
   [Process] int,
   [HostName] nvarchar(200),
   [Time] datetime,
   [TargetTable] nvarchar(200),
   [Action] varchar(10),
   [OldData] xml,
   [NewData] xml
)

 Here are the steps to fill the above table from the parsed message received from SQL server service broker:

1. First of all we will we cast the message body to XML format as follows:
   
   DECLARE @PayLoad xmlWAITFOR (RECEIVE TOP(1) @PayLoad = message_body FROM NotificationQueue), TIMEOUT 5000
   
   
2. Then we prepare the message payload by letting SQL server parse the message and convert it to parsed XML document ready for consumption:
   
   DECLARE @idoc intEXEC sp_xml_preparedocument @idoc OUTPUT, @PayLoad
   
   
3. Now I will use OPENXML to open the document and start processing it by parsing the message header into SQL data format and insert the data to our audit trace table as follows:
   
   insert into  AuditTrace ([User],[Process],[HostName],[Time],[TargetTable],[Action]) SELECT * FROM OPENXML(@idoc, N'//NotificationMessage/Header') 
   with ([User] nvarchar(200),Process int,HostName nvarchar(200),Time datetime,TargetTable nvarchar(200),MessageType varchar(10))
   
   
4. By using XQuery I will update our previously inserted row in the audit trace to amend the old and the new changed data:
   
   Declare @Key bigint = SCOPE_IDENTITY()
   Update audittrace set NewData = @PayLoad.query(N'//NotificationMessage/New'), olddata = @PayLoad.query(N'//NotificationMessage/Old')
   where AuditTraceid = @Key
   
   
5. The last thing to do here is to clean up the memory to close the previously opened XML document:
   
   EXEC sp_xml_removedocument @idoc
   
   
   

 By completing these tasks we will have a audit data in our audit trace data table like screen-shot below :

I want to consider another situation where we want to audit the changed data to another table in our database that has the same schema as the production table but is suffixed with (Audit), I could do this by using complex XML with some SQL server temp tables and some loops, instead I will use SQL CLR to achieve my needs, The common language runtime (CLR) integration feature is off by default, and must be enabled in order to use objects that are implemented using CLR integration. To enable CLR integration, use the clr enabled option of the sp_configure stored procedure:

sp_configure 'show advanced options', 1;
GO
RECONFIGURE;
GO
sp_configure 'clr enabled', 1;
GO
RECONFIGURE;
GO

To create our CLR stored procedure, open Visual Studio 2010 and create new database project, choose your preferred language (I will take C#) as you can see below:

After creating the project you will need to choose your database in which deployment will take place, then add a new stored procedure which will contain our parsing and auditing code as follows:

Build your project then click deploy which will deploy you assembly and creates a new stored procedure to your target database chosen earlier. Now let's move to SQL server, first we will need to create a new stored procedure that will receive notification message from server broker and passes it to our CLR procedure for auditing:

The last thing we will need to do is to alter our service broker queue to use internal activation as follows:

ALTER QUEUE NotificationQueue
    WITH ACTIVATION
    ( STATUS = ON,
      PROCEDURE_NAME = NotificationQueueActiviation,
      MAX_QUEUE_READERS = 20,
      EXECUTE AS SELF
    );
GO

Now if we run any CRUD operation against our database then we will have our log as you can see below:

You can find the SQL server CLR project which contains a simple parsing and auditing mechanism here. AuditProc.zip (25.18 kb) 

Service Broker helps developers build asynchronous, loosely coupled applications in which independent components work together to accomplish a task. These application components exchange messages that contain the information that is required to complete the task. Service Broker's features provide a number of significant benefits to database applications. These features and benefits include:

• Conversations: Service broker is all about messaging, two entities can exchanges messages without caring about the low level communication infrastructure; conversation is a reliable, persistent communication channel that guarantees message delivery.
• Message ordering and coordination: Service Broker queues are integrated into the database which means regular database maintenance and administration also included and handles the most difficult tasks involved in writing messaging applications. These difficult tasks include message coordination, reliable message delivery, locking, and starting queue readers.
• Transactional asynchronous programming: Message delivery between applications is transactional and asynchronous; if your application transaction rolls back, all Service Broker operations in the transaction roll back. In asynchronous delivery, the Database Engine handles delivery while the application continues to run.
• Support for loosely coupled applications: SOA is about messaging and service broker is built using messaging infrastructure so it supports loosely coupled applications. Your applications do not need to run at the same time and do not have to know the physical location or the implementation of the other participants in the conversation.

The following illustration presents a high-level view of Service Broker network communication between two SQL Server instances:

Notice that the conversation is a persistent, logical connection. The conversation can occur over any period of time, and during that period of time, the conversation can use any number of network connections.

Network connections occur between two Service Broker endpoints. These connections use TCP/IP. If the connection is inactive for a short time, SQL Server closes the network connection. To deliver a message, Service Broker holds the message in the transmission queue for the database that sent the message. The recipient delivers the message directly to the queue for the destination service. The queue for the sending service is not involved in the operation.

Let's now but service broker into practical example which is  a generic database CRUD auditing. The first thing you will need to do is to enable service broker for your database and this could be done by using the following SQL command:

alter database [DataBaseName] set enable_broker

This option activates Service Broker message delivery, preserving the existing Service Broker identifier for the database. if you have an old database that uses service broker then you will need to generate a new identifier for your service broker:

alter database [DataBaseName] set new_broker

Now it's time to create our service broker objects, primary I will create a template message type to hold the actual data payload for exchange during a certain conversation. A message type defines the name of a message and the validation that Service Broker performs on messages that have that name:

CREATE MESSAGE TYPE [//intellecting.net/Notification] VALIDATION = NONE ;

Note that the message type specifies a validation type of NONE because the message will contain data that is not well-formed XML or it might binary data.

Then I will create service broker contract which defines the name of a specific business task and list the message types used in that task. Service Broker contracts define two different service roles: the initiator and the target. The initiator of a conversation begins the conversation by sending a message to the target. The contract that the conversation uses defines which service role can send messages of a given message type:

CREATE CONTRACT [//intellecting.net/INotification] ([//intellecting.net/Notification] SENT BY INITIATOR);

The above SQL command indicates that only the initiator of the conversation can send messages of the specified message type which means that we defined a one way contract.

Now I will create a service broker Queue which stores messages, when Service Broker receives a message for a service, Service Broker inserts the message into the queue for that service. Service Broker manages queues and presents a view of a queue that is similar to a table:

CREATE QUEUE NotificationQueue WITH STATUS = ON;

A queue may be associated with a stored procedure. In this case, SQL Server activates the stored procedure when there are messages in the queue to be processed. SQL Server can start more than one instance of the stored procedure up to a configured maximum as shown below:

CREATE QUEUE AuditQueue WITH STATUS = ON,
      ACTIVATION (
            PROCEDURE_NAME = AuditMessage,
            MAX_QUEUE_READERS = 20,
            EXECUTE AS SELF )

As you can notice from the above SQL code that the activation stored procedure is defined with a maximum of 20 instance and will execute as the current user.

If your auditing database will lay on the same server with the OLTP database then it is preferable to use  internal activation with stored procedures as it is a common way to design Service Broker applications and of course your stored procedure will contain the routine needed to audit your data changes.

Finally we will need to create a service which represents a business process as a distinct set of tasks, each contract within the service represents a specific task. Each service uses a queue to store messages, messages sent to the service are delivered to the queue:

CREATE SERVICE NotificationService
       ON QUEUE NotificationQueue
      ([//intellecting.net/INotification])

If your auditing database will be on another SQL server then you will need to configure your service broker instance to use message routing, initially we will create service broker endpoint that exposes our service capabilities to the outside world:

After configuring the endpoint, you must set up security between the distributed Service Broker services. By default, Service Broker doesn’t allow two Service Broker services to communicate without configured security. For transport security I will use Windows Authentication especially if the two service broker instances are registered in the same domain and  It would even work if you have a trusted relationship between two Windows domains.

You need to run your SQL server instance with a certain domain account, for example intellecting\ahmedamir, you can do this easily through Services MMC snap-in. you now have to create SQL Server logins for the SQL Server that communicates with you. Note that this login represents the service account of the remote SQL Server machine, not the service account of the instance where you’re creating the Service Broker endpoint:

CREATE LOGIN [intellecting\AuditAccount] FROM WINDOWS;

You will need to grant connect and send permission for your service as follows:

GRANT CONNECT ON ENDPOINT::AuditBrokerEndpoint TO [intellecting\AuditAccount];
GRANT SEND ON SERVICE::[NotificationService] TO PUBLIC; 

After configuring the endpoint and service security, we need to instruct Service Broker to route our sender messages to a certain SQL server instance, so on the sender service broker instance we will configure our routing as follows:

By creating all service broker objects; our messaging infrastructure is ready for message exchange through a single SQL server instance our by using multiple SQL server instances. Now let us try to send message to our service broker, I will create the service initiator; the following stored procedure opens a conversation to our service and submit the XML message to the service queue:

      END CATCH
END

As can be observed from the above code that no message encryption is used and the procedure sends a message to a local service queue, if you want to send a message to another SQL server instance then you will just use the name of the service in the Service Broker Route defined earlier.

The question now is how to format your audit message with the hooked data and how to use the above stored procedure to send your audit messages to service broker. In order to hook any data changes you will need to create SQL AFTER trigger to capture data changes, you can create three trigger; for insert, Update and delete but I will consolidate them to one trigger as follows:

As can be noticed from the above code that our trigger is divided into multiple parts:

1. In the first part, we dynamically detect the changed columns of target table by calling SQL server trigger function 'COLUMNS_UPDATED()' which in turn returns a bitmask of the changed columns according to column order, then we pass this bit mask to system function to get the names of the changed columns and finally we format the result by using XML.
2. In the second part, we detect the trigger action [Insert, Update or Delete] by selecting the number of rows in the INSERTED & DELETED tables and according to the trigger type we format the data changes by using XML.
3. In the third part, we collect information about the user and actions to be our message header and then we combine all the formatted data generated by the first two steps to produce the final message to be audited.
4. In the final part, we just submit the formatted XML message to our audit procedure which in turn sends the message to service broker infrastructure.

Now I can say that we finished the first part of creating our auditing system, the remaining is how to get your XML messages from service broker (in case of distributed brokers) and how to parse them to produce our audit data. I will introduce message receiving and parsing in my next talk soon.

I attached the SQL server database that contains all the code discussed here. DATA.zip (264.13 kb)