A couple of months ago I implemented an alert for host throttling on one of our BizTalk 2006R2 servers. I added an alert to the event log as described by http://www.deves.net/blog/post/Biztalk-2006-Host-Throttling-Performance-Monitoring-Alert.aspx. One day i was surprised to find this alert in the event log repeating every couple of seconds.
Event Type: Information
Event Source: SysmonLog
Event Category: None
Event ID: 2031
Date: 15/04/2010
Time: 12:25:34 a.m.
User: N/A
Computer: ABizTalkServer
Description:
Counter: \\ABizTalkServer\BizTalk:Message Agent(BizTalkServerApplication)\Message delivery throttling state has tripped its alert threshold. The counter value of 4 is over the limit value of 0.
How to BizTalk Server Implements Host Throttling told me that a message throttling state of 4 means that the process memory has exceeded a specified threshold. I switched on performance monitor and measured the following counters. eeeK!
I restarted the BizTalkServerApplication host instance and the memory usage dropped to an more acceptable level 75MB and the throttling alert disappeared. It took me a while to work out why Process memory usage threshold is about 330Mb. The Process memory usage for the host was set at the default namely 25% .The ABizTalkServer had 2Gb of Addressable Memory and was 32 bit Windows. This means that the practical limit for virtual bytes is only 1400MB.
Why is this host consuming a lot of memory? I decided to collect the following performance counters over the next day to see if the memory usage was increasing;
- Process/ID
- Process/Private Bytes
- Process memory usage
These articles suggest I should have added a few more counters; How to troubleshoot a memory leak or an out-of-memory exception in the BizTalk Server process and Debugging Native memory leaks with Debug Diag 1.1.
To work out which Process/ProcessID belongs to the BizTalkServerApplication Host run the following from the command line;
tasklist /S AbizTalkServer /svc /fi “imagename eq btsntsvc.exe”
The output should something look like this
Image Name PID Services
========================= ======== ============================================
BTSNTSvc.exe 6732 BTSSvc$BizTalkServerTrackingOnly
BTSNTSvc.exe 10460 BTSSvc$BizTalkServerApplication
BTSNTSvc.exe 12580 BTSSvc$BizTalkCRM
Now the Process/ID equal 6732 is BTSNTSvc#7. This is the process instance that corresponds to the BizTalkServerApplication Host service.
The monitor trace showed a gradual increase in private bytes for BTSNTSsvc#7 and process memory usage for the BizTalkServerApplication during the day. All the other BTSNTSsvc processes remained at about the same level. This means one of the components running on the BizTalkServerApplication has a memory leak.
I choose to use the DebugDiag tool as described in the articles above to pinpoint the component that was leaking. I collected many dumps and analysed them but was perplexed because all the reports were missing the “call stack trace” that would allow me to see what was happening for the main allocators. I finally realised that i need tick the box to record call stacks on the options and settings page to get the stack traces. The default install of DebugDiag leaves this option turned off. Once this option was turned on the stack trace pinpointed the component that was leaking . Now I could to roll up my sleeves and fix the memory leak.
