One of our administrators observed this error after adding a new SFTP receive location and this article describes my analysis and solution to this problem.
The Receiving host was consuming over 2Gb of memory when viewed by the task manager and Process Memory counter was close to the Process Memory threshold.
A quick check showed that the only BizTalk artifacts running on this host where SFTP adapters, pipelines and maps. As an aside the new BizTalk 2013 SFTP adapter had been one of the selling points to this customer when we implemented BizTalk Server.
What was so special about this new SFTP receive location that had been added? Answer not much because it is was very vanilla , only using a pass through pipeline and with the default settings supplied by Microsoft for polling.
A performance monitor data collection was set up to collect some of the BizTalk performance counters and the Receiving host was restarted. The process memory for the Receiving host ( the highlighted black line in the chart) quickly reduced to less than 300Mb but then slowly started to rise over time and shown in the graph below. I have included the same counter for the sending host ( the light blue line) as a comparison. After 24 hours the receiving host had increased to over 800Mb. The chart shows the number of messages and the receiving rate , red and green respectively. This shows that all the messages are only received between 5 a.m and 8.30 a.m each day. The memory is increasing on the Receiving host all of the time regardless of whether messages are being processed or not.
What is the difference between the new SFTP receive location and the old ones that is causing it to consume so much memory? The administrator discovered that the other locations where configured to poll every 10 minutes instead of the default 5 second default poll and ran on a schedule between 5 a.m and 9.00 a.m. They immediately reconfigured the new SFTP receive location to also poll every ten minutes and only run between 5 a.m and 9 a.m. This change is marked on the graph above. The rate of increase of process memory consumption on the receiving host decreased immediately and might have even flat lined.
Conclusion
If you use the BizTalk 2013 SFTP receiving adapter you should be careful to limit its polling frequency by configuring the polling interval to be in minutes rather than seconds and to only run during scheduled times.
The feedback that i would give the Microsoft Product group about the SFTP adapter is that they should;
- Add the configuration of a proxy because that is what most of my customers want.
- Add logging similar to the old classical FTP adapter.
- Change the default polling interval from 5 seconds to 5 minutes. It is insane to running a FTP get that quickly.
- Check the SFTP receive adapter for a memory leak.
