This blog will describe how I replaced a BizTalk SFTP receive adapter with an Azure connection. Following on from a previous post I present an alternative hybrid integration pattern for SFTP polling. An Azure Logic App and a hybrid connection to BizTalk server replaces the BizTalk 2013 R2 SFTP adapter.
SFTP polling with the BizTalk 2013 R2 SFTP adapter
Shaktour shows how to use the BizTalk SFTP adapter here, which is new in BizTalk 2013. BizTalk 2013 R2 contains some upgrades and cumulative updates for this adapter. Starting with BizTalk Server 2016, the SFTP adapter uses WinSCP to connect to SFTP, and therefore supports a larger range of SFTP servers. The comments and problems below only apply to the BizTalk 2013 R2 SFTP because the BizTalk 2016 SFTP adapter is clearly an improvement on the early 2013 versions.
The BizTalk SFTP adapter that I am going to replace was configured like this.
In my hands the BizTalk 2013 R2 is problematic with some SFTP servers. The issues are;
- The event log shows many warning like “The adapter “SFTP” raised an error message.”Microsoft.BizTalk.Adapter.SftpInvoker.SftpException: Unable to initialize SFTP connection. Ensure that the specified server’s configuration details are correct”.
- Some files stick in the SFTP folder.
- SFTP locations stop and must be restarted.
- The SFTP server periodically kills all connections for 600 milliseconds to free up unused connections. The BizTalk receive location has to be restarted when this occurs.
With this in mind I turn to Azure to see if it can do better.
SFTP Polling with Azure
The plan is to replace the BizTalk SFTP receive adapter with a BizTalk SB-Messaging adapter that polls an Azure Service Bus. An Azure SFTP connection polls the SFTP folder location and adds the messages to the Service Bus. This isolution is suitable for those of us running BizTalk Server 2013 R2 Standard.
If you are using BizTalk 2016 you might look at using the Azure On-premises Data Gateway instead of the SB-Messaging adapter. That hybrid integration solution requires that you have an integration account as well as the latest version of BizTalk. If you don’t already use an Azure integration account this could be a cost barrier to overcome. For the moment this discussion is restricted to what we can do with BizTalk 2013 R2 only.
First SFTP Polling Attempt
My first attempt uses an Azure SFTP trigger and follows a Microsoft example. In this example, the SFTP – When a file is added or modified trigger initiates a logic app workflow when a file changes on an SFTP server. This example requires no code, only configuration. Secondly the “Send Message” to service bus action use SFTP filename as Session Id. As a consequence when BizTalk retrieves the message from the service bus, the SFTP filename appears in the context of the message as SessionId. Thirdly a SFTP Delete action removes the file from the SFTP server using the SFTP file identifier.
Finally a BizTalk Service Bus Messaging receive adapter retrieves the message from the Service Bus. I followed Stejan Wiggers classic SB-Messaging example but I used SAS token instead of the deprecated ACS token. Once again this requires no code, only configuration.
Unfortunately while the first run works seamlessly, subsequent runs failed to trigger when new files are added the SFTP adapter. In our hands the SFTP – When a file is added or modified trigger does not work. This might be because of the type of SFTP server I work with but if you have managed to get the SFTP trigger to work please let me know how.
Second SFTP Polling Attempt
Working around this issue;
- a recurrence trigger replaces the SFTP trigger.
- Secondly list the files using the “SFTP list files in folder action”.
- Thirdly a “For Each Loop Action” iterates over each DisplayName, which contains the file name.
- Fourthly a conditional action checks the file name contains a xml extension.
- If the condition is true retrieve the file using the Path in a “SFTP Get content using path” action.
- Add the file to a Service Bus using the “Service Bus Send Message” action.
- Finally delete the file using the Path in a “SFTP Delete file” action.
This time Logic App continues to run and pick files from the SFTP folder, passing them to the service and BizTalk Server picks up the files from the Service Bus . It was interesting to see that sometimes one of the files in the list failed to pick but are picked up in a subsequent run. I think this is because of the fragility of the SFTP server that I am using.
Costing of Hybrid SFTP Polling
We currently do not have an Azure subscription and will host this in a pay as go Azure subscription to firmly establish costs before looking at other options. I have used this as the basis of my estimates. The final Logic App contains 1 trigger and 6 actions. I calculated the cost per month conservatively at $NZ37 per month. I consider this good value because we have to restart this adapter at least 5 times a week,
Future Plans
After deploying this to production we will determine whether Azure SFTP polling hides some the failings of the BizTalk 2013 R2 adapter. Initial results in development are promising because we do not have to restart the SFTP connection regularly.
This hybrid migration sets us up well to move the entire interface into Azure by extending the Logic App to do the downstream mapping and transport protocol mediation. The only thing that is stopping us doing this is the cost of an integration account. Why do we have pay for an integration account if we have BizTalk?
The SFTP adapter has changed in BizTalk 2016 and we look forward to seeing if this solves some of the issues we reported in the introduction. When we upgrade to BizTalk 2016 we will switch back to see if this solves the issue discussed above.
Conclusions
This post has shown an example of moving a troublesome BizTalk adapter into Microsoft Azure, with no extra coding. There is an extra cost but when we weigh that up against the time required to restart the BizTalk receive adapter it is well worth it.
