I was asked to help today when a new BizTalk 2013 server just stopped processing messages. I have decided to talk about it here because there are some good lessons can be learnt. Nothing was happening on the server but there was about 5000 active instances just sitting in the message box. After a while the analyst decided to restart the host instances and the BizTalk server got very busy and processed all the message in less than a minute.
The analyst then told me that this was happening ever morning and asked what they could do about it . Every morning two files are received at about the same time that contain about 10,000 records each. These individual records are de-batched, processed by BizTalk server and sent out on two send ports. The analyst also volunteered that a couple of days ago he had seen the following warning in the event log, “BizTalk host Sending throttled because InflightMessageCount exceeded”. He suppressed the warning by increasing the Inflight message setting on the receiving host from 75% to 300%.
Now I knew what the matter was! Throttling kicks in on a BizTalk server to ensure the server keeps running no matter what is happening and is key part of achieving reliability and guaranteed delivery of messages. The server was exhausting all of its resources and then processing was stopping all together. Throttling ensures that there is always enough resource for BizTalk server to keep going albeit very slowly. our analyst hero had just turned throttling off allowing the server to get into a resource starved state!
I suggested that the analyst changed the inflight message setting back to the default of 75%. The next morning the analyst was grinning from ear to ear because the BizTalk server had not stopped processing in the morning but the throttling warnings appeared in the logs again. I explained that this was good because throttling was kicking in to make sure that none of the resources on the server were being exhausted. I also made him aware that this was a warning message not an error.
Now what exactly is going on. I had also asked the analyst to capture some performance counters on the server while the errant processes were running and some of these are shown below.
The receiving host throttles first with a Publishing throttling state of 2 (the yellow line). The “How BizTalk Server Implements Host Throttling” article on MSDN explains what is happening. The high number message being published to the message box in a short period of time causes a load on the server to increase. Throttling blocks the publishing thread to ensure that the message publishing incoming rate matches the message publishing outgoing rate ( the red dotted lines). This is good because means the server never runs out of resource while the messages are being published to the message box.
Secondly the sending host starts throttling with a delivery throttling state of 1( the blue line). The high number of messages being received by the send port from the message box in a short period of time causes the load on the server to increase. This time the throttling blocks the delivery thread to ensure that the delivery incoming rate matches the delivery outgoing rate(the light blue and green line). This is good because means the server never runs out of resource while the messages are being received from the message box by subscribers.
Finally the sending host starts to throttle with a publishing throttling state of 2 ( the green line). The high number message being published to subscribers in a short period of time causes a load on the server to increase. Throttling blocks the publishing thread to ensure that the message publishing incoming rate matches the message publishing outgoing rate ( the pale blue dotted lines). This is good because means the server never runs out of resource while the messages are being published to the message box.
In summary throttling is working how it should because the message incoming rates are matching message out going rates. I don’t think that there is any host configuration that can improve the performance of this badly written application. Throttling will make sure this errant application does not affect other processes running concurrently on the same BizTalk server. The only thing our analyst hero can do is to go back to the developer and ask him to redesign his solution such that it performs better.
