BizTalk Server - Performance Tuning

While considering performance improvement of a BizTalk Server or a BizTalk application, steps needs to be taken while writing your code as well as while setting up the BizTalk Server Environment along with the application deployment. Below are some points one should consider while setting up the BizTalk Environment.

1. Verify that BizTalk SQL Server Agent Jobs are up & running

BizTalk Server includes several SQL Server Agent jobs that perform important functions to keep your servers operational and healthy. You should monitor the health of these jobs and ensure they are running without errors. One of the most common causes of performance problems in BizTalk Server is the BizTalk Server SQL Agent Jobs are not running, which in turn can cause the MessageBox and Tracking databases to grow unchecked. One should verify that the BizTalk Server SQL Server Agent jobs are completing in a timely manner without any errors.

2. Define auto-growth settings for BizTalk Server databases to a fixed value instead of a percentage value

Database auto-growth should be set to a fixed number of megabytes instead of to a percentage (specify file growth In Megabytes). This should be done so that, if auto-growth occurs, it does so in a measured fashion. This reduces the likelihood of excessive database growth. The growth increment for BizTalk Server databases should generally be no lower than 100 MB.

3. Even Distribution of Host & Host Instances 

On the production environment, one should take every necessary step to distribute the hosts & respective host instance across multiple servers available in the BizTalk Server Farm in such a manner that one host should not be highly loaded and load should be equally distributed accordingly in order to achieve good performance of the application & BizTalk server.

4. Complete & Proper Deployment on Production Server 

In case of multiple server BizTalk Farm, you should take care that every application should be completely deployed on BizTalk and properly installed on all the servers in the farm along with all required settings (BizTalk Config file, registry, SSO, any xml configuration, etc.)

5. Move the Backup BizTalk Server output directory to a dedicated SAN/NAS 

Move the Backup BizTalk (Full and Log backup) output directory to a dedicated SAN/NAS, and then edit the Backup BizTalk Server job to point to the dedicated SAN/NAS. Moving the Backup BizTalk Server output directory to a dedicated SAN/NAS will reduce disk I/O contention when the job is running by writing to a different disk than the job is reading from.

6. SQL Server Max Degree Of Parallelism (MDOP)

 The Parallelism setting governs the number of processors on which any particular Transact-SQL statement can run at the same time. If the Parallelism setting is set to use one processor, the SQL Server query optimizer will not create execution plans that permit any particular Transact-SQL statement to run on multiple processors at the same time. Max Degree of Parallelism is set to “1” during the configuration of BizTalk Server for the SQL Server instance(s) that host the BizTalk Server MessageBox database(s). This is a SQL Server instance-level setting. This setting should not be changed from this value of “1”. Changing this to anything other than 1 can have a significant negative impact on the BizTalk Server stored procedures and performance.

                                                                     

7. SQL Server Statistics on the Message Box Database 

Updating statistics ensures that queries compile with up-to-date statistics. However, updating statistics causes queries to recompile. The following options are turned off by default in the BizTalk Server MessageBox database when it is created:

a.       Auto Create Statistics

b.      Auto Update Statistics

Do not enable these options on MessageBox databases. Enabling the "auto create statistics" and "auto update statistics" options can cause undesirable query execution delays, especially in a high-load environment.

8. Setting the “text in a row” table option on specific MessageBox database tables 

SQL Server provides a table option called text in row to declare that the contents of the fields of type text, ntext, or image data whose dimensions are smaller than those of a data page (8Kb) must be stored in a data row. By setting this option on BizTalkMsgBoxDb tables (Parts table and Spool table), you can increase message throughput when working with small messages which have a small context and orchestrations that have a small persistence size.

Introduction to REST

Two weeks back when I started reading about REST, there were thousands of articles available on Internet and after going through quite a many of them I finally found myself at the same position where I started. All of them were very informative but confusing as there were so many terms used like resources, verbs, nouns, blah blah blah…. I then decided to note down each term so that I can relate terms with each other while moving forward and gathered all the information below.

While working with traditional web services, we generally create web methods and expose them to our clients for their use. These web methods expects some kind of input from the client (in terms of string, integer, xml, etc.) in order to meet the required expectation.

Imagine a traditional web service which does not expose methods but exposes objects. An object does not have any input or output, which does not do anything because it is a NOUN not a VERB, a THING not an ACTION.

Let’s take an example of an automobile manufacturer ‘Volkswagen’ which exposes a method called ShowCarDetails( ). This method expects the car name as an input and gives the car details as output.
Think for an instance, in place of this web service, there is a new one which exposes car names as objects. So, when a client looks at it, instead of ShowCarDetails (), he will see POLO, VENTO, JETTA, PASSAT, etc. as objects. These objects don’t have any methods written for them to be invoked by clients. These objects (NOUNS) need to be acted upon with some action on them (VERB). In this case, in order to fetch the car details we will apply the GET (VERB) action on POLO (NOUN) to retrieve its details.

There is a term in OOPs called “Polymorphism” which says that a single verb can be applied to multiple nouns. This was the biggest challenge to identify the common verbs (actions) which can be applied to all the nouns (objects).  HTTP was the answer for this. HTTP is a "Protocol" which says: a client can send a request to a server which is either a GET or a PUT or a POST or a DELETE (there are other verbs as well – HTTP 1.1 supports 8 verbs). The entire web called as INTERNET works on these verbs only. So, if a web service that exposes only objects supports these four operations "on its exposed objects", any client should be able to do anything with that web service! This is what is called as REST.

REST is basically a term introduced by Roy Fielding in his Ph.D. dissertation to describe an architecture style of networked systems. REST is an acronym standing for Representational State Transfer.

If we go by the definition:

“REST exposes resources (sources of specific information), each of which is referenced with a global identifier (e.g., a URI in HTTP). In order to manipulate these resources, components of the network (user agents and origin servers) communicate via a standardized interface (e.g., HTTP) and exchange representations of these resources (the actual documents conveying the information).”

In many ways, the largest current world Implementation of REST is the INTERNET. RESTful applications use HTTP requests to post data (create and/or update), read data (e.g., make queries), and delete data. Thus, REST uses HTTP for all four CRUD (Create/Read/Update/Delete) operations.

REST is not a standard. One can only understand it and design his web services in that style. While REST is not a standard, it does use standards:

• HTTP
• URL
• XML/HTML/GIF/JPEG/etc (Resource Representations)
• text/xml, text/html, image/gif, image/jpeg, etc (MIME Types)

Now, going by the same example as used above, if we need to get details of Volkswagen car POLO we can get it by just using a URL as shown below (as an example):

"http://www.volkswagen.com/POLO"

When you will type this URL in browser, it will respond back to you with the required details in the pre-defined format as set up in the service (xml, text, json, etc.). Please note that the above URL will not actually work as this does not exist, I have used it as an example.

Much like Web Services, a REST service is:

• Platform-independent (you don't care if the server is Unix, the client is a Mac, or anything else),
• Language-independent (C# can talk to Java, etc.),
• Standards-based (runs on top of HTTP), and
• Can easily be used in the presence of firewalls.

Like Web Services, REST offers no built-in security features, encryption, session management, QoS guarantees, etc. But as with Web Services, these can be added by building on top of HTTP:

• For security, username/password tokens are often used.
• For encryption, REST can be used on top of HTTPS (secure sockets).
• ... etc.

One thing that is not part of a good REST design is cookies: The "ST" in "REST" stands for "State Transfer", and indeed, in a good REST design operations are self-contained, and each request carries with it (transfers) all the information (state) that the server needs in order to complete it.

Conclusion:

REST is an architectural style which says to expose resources or objects to the consumers of the web service and allows the basic HTTP operation to be performed on them. In the next part of an this article, we will see detailed description about REST and how to create your first REST service.

References:

Representational State Transfer

http://www.xfront.com

About Rest Services on CodeProject.com

REST Tutorial by Dr. M. Elkstein

http://rest.elkstein.org

REST and the Real World

( http://www.xml.com/lpt/a/2002/02/20/rest.html )

EAI Blues | Developer Vs Tester

After a couple of hard core technical entries, here is one which although is true in its own nature, but not heavy on the mind.

We all know and agree that Development and Testing teams are not always kind to each other. Till date, the reasons are unknown but everyone agrees that yes, there is a cold war.

In projects implementing EAI, Middleware solutions include numerous applications, end to end scenarios, business rules, multiple use cases, so the software testing part is truly a bigger challenge than development.

The main reasons resulting in multiple config defects, mapping defects, design changes, deployment failures can be attributed to the differences and absence of synergies between these two teams.

Testers keep opening defects on documents, test scripts, deployment manuals, mappings, result sets and keep complaining that the developers have not done a thorough job. Developers keep on saying the testers are a bunch of idiots who have little functional knowledge and do nothing but raise useless defects.
Amidst all this chaos, the issue is, that key functionality and business logics are often missed and defects arise in SIT or end to end or full data set testing.

So now, what can we do to avoid these defects in the first place? The answer according to experienced program managers is to treat both the dev and test team with equal importance and like two little brothers fighting for the candy. You should always pay equal importance to map the requirements of the testers and deliverables of the developers.

Furthermore below are some of the best practices that we can follow to ensure a 'Fast and Flawless' (a quote taken from my current company) delivery.

• Involve resources from the testing team right from the beginning. It is better if they understand the Interface design as well, and like everyone always say ' See the Big Picture '.
• They should have an approved test catalog by the time Interface design is complete. This is will help a lot while creating the test scripts and expected results.
• Developers and Testers - Both these teams should do a cross review. Developers should get their deployment manuals and unit test cases reviewed by the Testing team, but the Testers should get the test scripts and expected results verified with a Business Analyst and NOT the Development team.
• Developers tend to have a blind eye towards the result set. If the Interface has run fine, data has been published, there are no errors in the Logs, they WILL (I can tell you as I have been in development for 5 years) mark the test as passed without verifying if the data entered is absolutely correct. So ask the Developers to create expected results before unit testing, NOT after. (I have seen developers copying Interface execution results into expected results!!!!!)
• Keep the documents UP TO DATE. People NEVER do it, big project, small project, big company, small company, 10 yr experienced, 1 yr experienced. Developers and Testers both tend to finish the work at hand, before they update the relevant documents which often results in the documents never been updated. The Program Manager should always and always make sure that the info in the repository is up to date.
• Big or small, document or code, please please track all defects and make sure to analyze those defects on regular intervals to incorporate into best practices and guidelines.

That being it on a crude level, the talks are endless, viewpoints to the tunes of stars in the universe and standards like alien languages. There is no set rules for success, only a few guidelines and best practices that might help you reduce the risk and increase quality.

Thank you for your time and please do post your valuable comments to make the post and the blog more helpful.

Achieving Messaging Only scenario through Web Services without deploying the receive side web service on BizTalk IIS

In many systems, when we come across a requirement of receiving the request into BizTalk from External Sources through SOAP adapter (at receive side), we generally create a web service for receiving the request and deploy that service in BizTalk IIS. This web service then publishes the input message into the BizTalk MsgBoxDB and then finally the message arrives at SOAP receive location and moves on for further processing. On the send side, there is the actual web service where the message needs to land.

In order to avoid this actual web service creation and deployment on BizTalk IIS, we can use the WCF service behavior feature. To achieve this we only need to make sure is that the input schemas which send side web service is expecting should be deployed on BizTalk.

A graphical representation of the actual scenario is:

Below are the steps to accomplish this: 

1. Open the BizTalk admin console and create a new application “SampleApplication”

2. Create a new one way receive port named “ReceiveRequest”

3. Create a new receive location named “ReceiveSOAPRequest”. Select the transport type as “WCF-Custom” and XMLReceive pipeline.

4. Click on Configure tab which will open the WCF-Custom Transport Properties. On the General tab, type the Address which you want to be browsed (exposed to source) from outside by clients. I have written it as http://<YourServer>/MathServiceTest/MathServiceTest.svc. You can use anything you want.

5. On the Binding tab, select the binding as “basicHttpBinding”.

6. On the Behavior tab, right click on “ServiceBehavior” and Add extension. Then select the “serviceMetadata” from the list.

7. Click on the “ServiceMetadata” on left panel which will show you its properties on right panel. Put the service URL in the “externalMetadataLocation property”. This URL is the one whose input schema is deployed in BizTalk (which is be further invoked) which would look like http://<ServerIP>/BizTalkWcfServiceSchema/Service1.svc. Also, set the httpGetEnabled property to True. This is the destination web service where the message has to be sent.

8. Click OK. This will create a receive location as shown below.

9. Enable the receive location and restart the host instances.
10. Open internet browser and browse the URL which you typed in the general tab above i.e. http://<YourServer>/MathServiceTest/MathServiceTest.svc

Now, you will see that the URL you browsed is directly showing that service, one which you typed into the serviceMetadata.

So, you didn’t even bothered to deploy this MathServiceTest web service into BizTalk IIS and you are actually browsing it. When you pass any message to this service, it will directly land up onto BizTalk receive location and there on you can send that message further to the actual send location.

This may or may not be applicable to your scenario in particular. A front end web service may be required in places, where some business logic needs to be implemented before the BizTalk processing.

Benefits:

1. No requirement of developing a new Web Service for the messaging only scenario.
2. Lower number of Hops as the message directly lands on BizTalk receive location and is not routed through IIS.
3. Reduced Load on IIS.

Limitations:

1. As we are not writing an actual web service, there might be some concerns of Security.
2. May not be always applicable and feasible.

Please do comment if you have come across any scenario where this concept can be applied.

SOA and Web Services

Generally when we talk about SOA, the very first thing that comes to our mind is the web services. Too often, we equate a WebService and its implementation with the overall definition of SOA. In reality it is nothing more than an implementation. A good implementation, but just an implementation. A WebService is platform and language independent and its use of XML enables different platforms to communicate with one another. 

It is not true that every web service is SOA compliant and those Organizations/Companies which implements web services thinking they are implementing SOA, finally end with a total failure.

The world has been implementing SOA long before web services came into existence (you can refer to some case studies of SOA implementation at this link -- Case Studies). Below are few of many protocols which were used to implement SOA:

1. Advanced Program-to-Program Communication (APPC)

Advanced Program to Program Communication or APPC is a protocol which computer programs can use to communicate over a network. APPC is at the application layer in the OSI model, it enables communications between programs on different computers, from portables and workstations to midrange and host computers.

2. Remote Procedure Calls/Remote Method Invocation (RPC/RMI)

A Remote Procedure Call (RPC) is an inter-process communication that allows a computer program to cause a subroutine or procedure to execute in another address space (commonly on another computer on a shared network) without the programmer explicitly coding the details for this remote interaction. When the software in question uses object-oriented principles, RPC is called Remote Invocation or Remote Method Invocation.

3. Internet Inter-ORB Protocol (IIOP)

IIOP is an object-based protocol and has the potential to massively enhance the types of applications or services or databases that are built and communicate on the web. IIOP provides a comprehensive system through which live objects can request services from one another across the corporate networks over the internet or intranet.

In today's world, when everyone is moving to cloud, the best example of SOA implementation is what Cloud offers as:

A. Software as a Service (SaaS)

B. Platform as a Service (PaaS)

C. Infrastructure as a Service (IaaS)

So, we can say that Web services are just an implementation of SOA but not the only implementation. There are many more examples available on the internet about SOA implementations.