In this blob we ask the question “can you poll an FTP URL with two Logic Apps”? Back in the day of BizTalk server this was the conventional wisdom.
Do not configure multiple FTP receive locations to poll the same FTP URL. If multiple FTP receive locations are polling the same URL concurrently then each receive location can receive a copy of the file, which can cause data duplication. This behavior occurs because the FTP protocol has no provision for locking files when reading them from the target URL.
To provide high availability for the FTP receive adapter, you should configure the FTP receive adapter to run in a clustered BizTalk Host instance. See Considerations for Running Adapter Handlers within a Clustered Host.
Configuring the FTP Adapter – BizTalk Server | Microsoft Learn
I cannot find any guidance whether multiple Logic App FTP Connectors can poll the same location. Thus, I list my findings here.
Poll an FTP URL – Use Case
We have two identical Logic App deployments in the Australia Southeast region (ASE) and the Australia East region (AUE). Until recently ASE was cold. Enabling ASE only in a disaster situation. This is a classic active-passive DR architecture. Moving to active-active architecture by enabling all our Logic apps in both regions at once made us think about the above question.
FTP active-active scenarios
We deploy the same Logic App to two different to ASE and AUE regions and they have the same Logic App API connection. Thus, both Logic Apps poll the same FTP folder on a third-party server.
We observe that we have no issues with some FTP servers. However, for other FTP servers failures can occur because the Logic Apps in both regions pick up the same file. For example, when we delete the file eventually from the FTP server. Moreover, the same file can be processed twice, once in AUE and again in ASE. To tell the truth, duplicate processing is Ok for us because all of our workflows are idempotent, but it increases our Azure costs.
There seems to be no rhyme or reason why some FTP servers behave well but others do not. I attempted many solutions but to date have not found a good solution. I list my attempts below.
We consider two options;
- Only pick up from AUE at certain times and WEU at other times.
- or handle the failure on delete and get actions gracefully
If anyone knows of a better solution, please let me know.
Only pick up from AUE at certain times and WEU at other times
Firstly, forcing the AUE Logic Apps to poll out of step with the ASE ones was not successful. For example, if AUE polls at 0, 6, 12,18, 24, 30, 36, 42, 48, 54 ,60 minutes in the hour then the DR Logic app polls at 3,9,15,21,27,33,39,45,51,57 minutes in the hour.
What we did was configure a start time on one of the Logic Apps such that it polls out of step with the other one. Thus, disabling the one of the Logic Apps, setting the start time to a value halfway between the polling time of the other Logic App and finally enable it.
Although this works for a couple hours the trigger time drifts eventually such that both triggers are firing at the same time. If you set a polling time of six minutes, the trigger fires and picks up many files. The next polling time is six minutes after the last file. Not six minutes from the last skipped poll.
Secondly, we set a trigger condition on the AUE Logic Apps to only run if the minute part of the current time is even.
@equals(mod(int(utcNow('mm')),2),0)and a trigger condition on the ASE Logic App to only run if the minute part of the current time is odd.
@equals(mod(int(utcNow('mm')),2),1)
This works. We observe that when orders arrive at even minutes the trigger fires and the order are picked up.
When orders arrived at odd minutes the trigger was successful but the orders were never picked up from the FTP folder because it did not fire.
Nevertheless, we were worried that a trigger may never fire, and files would remain stuck in the FTP folder. Thus, we rejected this option and begun to explore handle the failures on delete and get Logic App actions gracefully.
Handle the failure on delete and get actions gracefully.
For each FTP Logic app trigger
- add error handling to gracefully stop the run when the error is about a file cannot be found, set the logic app run status as “Cancelled”
- setup a regular alert that fires when Cancelled status is more than X number, to catch a catastrosphe.
The pattern for all polling Logic Apps is
The steps are
- Add a scope action
- Move the three actions shown into the scope
- Create a condition to evaluate whether the Scope has failed or not.
result('Scope')[0]['status'] - Terminate with a status of Cancel if the Scope fails
- Else continue.
This also works. This costs more because there are more actions in the workflow. Furthermore, many Logic Apps have to be refactored.
Conclusions.
I have shown that configuring multiple Logic Apps to poll the same FTP URL can cause issues. If multiple Logic Apps are polling the same URL concurrently then each one can receive a copy of the file, which can cause data duplication and failures.
I have shown two solutions that work but both do not feel right to me. If anyone knows of a better solution, please let me know.
