What does this setting do? In this post I describe why I found this setting useful and ask what are the disadvantages are of turning this setting to on.
I have a BizTalk Server 2013 R2 that uses eight WCF-WebHTTP request response adapters with service bus relays that are hosted in the same isolated host. All of these receive locations use the same WCF Custom behaviours, BRE pipeline component, JSON encode pipeline component, JSON decode pipeline component that are configured in different ways. Some of the ports have maps configured and others have no maps configured. We have observed a memory leak on this server which causes some of the in–process and isolated hosts to throttle after a long time. The typical warning we see in the logs is “BizTalk host ,<hostname> throttled because ProcessMemory exceeded the configured throttling limit.”. These warning only go away when we brutally restart IIS. Stopping and starting the application pools or any of the in-process hosts does not fix the issue. We collected performance counters and then ran the PAL analyzer. What we found was a steady increase in memory used by the .NET CLR but it did seem to be specific to any host instance.
I reviewed all the code that i thought could be running on the receive locations and could not find any reason for a memory leak. I was at the point of running at running DebugDiag to get a dump to try to find the source of the memory leak when one of my colleagues, Leela Pachaiyappan spotted an error in the event logs that said
“There was a failure executing the receive pipeline: “Microsoft.BizTalk.DefaultPipelines.PassThruReceive, Microsoft.BizTalk.DefaultPipelines, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35” Source: “Unknown ” Receive Port: “CustomerAPI” URI: “/Service.svc” Reason: Attempted to access an unloaded AppDomain.”
I found this thread https://msdn.microsoft.com/en-us/library/aa577833.aspx that tells us how fix this permanently. The solution to this issue is “ by enabling Default application domain for isolated adapter in the Isolated hosts settings….The SOAP adapter runs in the IIS process space. If more than one Web service exists in the IIS AppPool, then every Web service ends up having its own AppDomain. By default all messaging engine objects are created in the first AppDomain (that is,. the AppDomain corresponding to the first Web service). If the first Web service is inactive for an extended period of time for any reason, IIS unloads the first AppDomain. When this happens, all services in the hosting process become unusable.”. I was really interested in this because I reasoned I was probably loading the same assemblies in many different AppDomains. If they were all loaded into the same AppDomain then the common assemblies would be shared and would this led to a decrease my memory usage. Furthermore would the spin up from idle be quicker because the assemblies for the services are always pinned in memory?
I had never heard of this setting before and started to research it. Other blogs have repeated this advice but no one spelled out what the disadvantages of switching this setting is. I asked around amongst my colleagues and the best they could come up with was “Normally having separate app domains provides isolation. In the .net world this provided a way to manage security between different layers in your application.“. I reasoned that this was not of concern to me and I decided to set the Default application domain for isolated adapter to true.
Once again we collected the performance counters and monitored the server over the next week. We did not observe any throttling because of exceeding ProcessMemory and the request response latency was marginally better. What was more surprizing was that we no longer observed a memory leak as shown below!
In summary I have found that enabling Default application domain for isolated adapter in the Isolated hosts settings has mitigated a memory leak on our BizTalk Server. The only disadvantage that I can see is that some assemblies are now permanently pinned in memory and occupy some of the memory on your server. What I cannot understand is why this setting is not set to true by default for every BizTalk Server Isolated host.
I wish to acknowledge Mike Howell for collecting and analysing all the performance counters.
