On thing about being a developer is that eventually your sins sometimes catch-up with you. Recently I got a call from a BizTalk server site that i had not worked on for over three years. They had a problem; all the orchestrations for one of their applications where dehydrating and not being rehydrated. No orchestration had completed from this application for over six hours and there were many dehydrated orchestration instances building up. They shutdown some of the hosts that this application was not using and then restarted the host that was doing the work for the application in question and immediately the 100 or so dehydrated orchestrations processed in a flash!
The question that was posed to me was why was this happening and how could I stop it happening again. The title of this blog gives the answer away but my first answer was that throttling must be occurring on their BizTalk server. The symptoms above are classic throttling behaviour which has been described here. I asked them if they had recorded the throttling state counters at the time of the incident because this would have help me diagnose the problem further as described here but unfortunately they no longer had this information.
OK. I connected to the site for the first time in three years and started to have a look. Wow they had been busy. They had added an extra host for every new application that they had created. They had over 20 applications. I ran up performance monitor to have a look at some of the BizTalk counters under the BizTalk:Message Agent. The first thing I did was to look at the Process Memory usage for each host which told me that there was no memory leak. I then noticed that the total current process memory for the hosts (about 1600Mb) is close to the amount Physical Memory usage (about 1800Mb). Thus it was very likely that the throttling had occurred because they had exceeded the available memory on the server. The customer added more memory to the server and they commented that the server had not run as well for a long time.
Now for the embarrassing part. I asked the customer why they had a host for every application. They replied because that is what you told us to do when you left three years ago. Oops! Since i had left whenever a new application had been created a new host had been created. As each new host had been created the total process memory usage had increased. I had to explain that this is not the recommended way to deploy BizTalk artefacts amongst hosts. I gave the customer the following guidelines to rationalize their host to reduce the number that they where using;
- Three hosts should be configured i.e.
- A receiving host containing all the receive handlers
- A sending host containing all the send handlers
- A processing host that contains all the orchestrations
- One host should be configured as a tracking only host.
- One host should be configured to contain all the receive handlers that run as isolated processes e.g. HTTP receive handler and SOAP handler.
- Extra hosts should only be added for any high priority tasks or to provide isolation of unstable components from the rest of the environment.
Thus a vanilla host deployment would contain five hosts which is a long way from the eighteen hosts that were deployed on the exchange.
What had I been thinking about three years ago? In those days ( BizTalk 2004 – ), I had been working with a lot of unstable BizTalk applications and I had got into the habit of deploying new applications to their own hosts to separate them from other badly running applications. Later on I learnt that this was not a good way to disperse BizTalk artefacts amongst hosts. I wonder how many other sites might be following this same bad practice.
