This is a tale of how to use Azure Devops to build, test and deploy BizTalk Applications. In particular it will focus on CI and CD for on-premise BizTalk 2016 Server.
Why am I using Devops?
Five weeks ago our CI/CD infrastructure stopped working. The Team City server for building our BizTalk solutions was deleted and the backups could not be restored. Now it gets worse, the Team city configurations were not saved anywhere either. It was a perfect storm for two other reasons. Firstly Alastair who had set up our build and deploy process had left that week. Secondly I became very ill and had an emergency operation. Returning to work after a week on sick leave i was all at sea because I was so reliant on on this to do any BizTalk development. The take home point is never rely on any DR plan unless it has been tested.
Alastair has been with us for a couple of years, quietly doing a great job with BizTalk. Owning the Supplier EDI ordering and implementing Devops practices in the team, his contribution was really good. I will miss his skills and professionalism. Unfortunately I had never dug deep into how he had set up our Devops process.
Thus I had to roll up my sleeves and recreate it all again. The next part of this blog will record how I set up a new CI/CD process. First I will describe the previous TeamCity build and then show how we moved this to Azure Devops.
Building BizTalk solutions with Team City
The old CI/CD process uses the following components;
- A Bitbucket source code repository
- TeamCity
- Visual Studio
- A BizTalk 2016 build and test server
- Specflow
- The BizTalk Deployment Framework (BTDF)
- Octopus server
Firstly, TeamCity pulls the latest source code from Bitbucket and builds the BizTalk assemblies. Secondly creating the deployment package using the BTDF. Thirdly creates a new release in Octopus. Fourthly deploys the octopus release to a test server. The build server uses Visual Studio 2015, Specflow and Octopus tentacles. Finally Specflow tests run to validate the deployment package. Check-in to the BitBucket repository triggers the whole process automatically and the results are notified to the developers.
Deploying to UAT and Production using Octopus occurs , only once the Specflow tests passed.
Examining my choices to rebuild the new CI/CD I choose Azure Devops instead of TeamCity. More importantly I chose this because most of our environmental configuration was in the BTDF or Octopus. Furthermore moving our on-premise deployment to here is more consistent with how we do our Azure deployments. Next, I will describe how you can setup such a CI/CD process for an on-premise BizTalk 2016 environment.
Building BizTalk Applications with Azure Devops
I have a confession I am primarily a user of CI /CD that other people have set up. Thus it was stretch finally do this for myself. What follows I hope is an interesting journey to you.
A quick search on how to do this came up with an article by Bill Chesnut and this is a great place to start. Subsequently I rang Bill because his article was a bit dated and he gave me some hints. More importantly he shared a task group that gave me the basics of what to do.
Next, I will talk through the important steps of our setup shown below.
- Create agent to build on the BizTalk server.
- Create service connections.
- Create a build pipeline.
- Create a generic task group for the build and octopus release creation.
- Create a release pipeline.
Devops – create an Agent Pool
To use our on-premise build server an agent pool, self hosted agent and agent services must be created. Indeed a BizTalk 2016 server is still required to build BizTalk solution because there are too many dependencies to run this on an hosted Azure pool. Furthermore BizTalk, 2016 solutions use Visual Studio 2015 and this will not be available as a hosted pool after March 23, 2020
With regards to this, the Microsoft documentation describes how to do this and carrying out this step was uneventful. Finally a new service runs on the build server as shown below.
Devops – Service Connections
Configure service connections for BitBucket and Octopus.
Devops – create a build pipeline
Creating an Azure pipeline the first step configures the name, self hosted BizTalk agent and Bitbucket repository.
A generic task group builds and creates the octopus release. This is used by all our BizTalk pipelines just replacing the parameters with different values.
Notably, I found one gotcha with build numbers because I was a newbie. Configuring the build number as $(major).$(minor).$(patch) .$(Build.BuildId) I found that deployments from Octopus did not uninstall the previous deployments. After much soul searching the root cause was that the BTDF only looks at the first three numbers in a build number and not four. Thus moving the $(Build.BuildId) to the third position solves this issue as shown.
Finally the build pipeline was triggered when a new build was triggered when a check in occurred for the solution.
Devops – Create a generic task group for the build and octopus release creation.
It is best practice to collect reusable tasks in a task group.
To start with a Nuget restore task runs. Secondly the BDTF is versioned.
Thirdly, a debug build is compiled using Visual Studio 2015.
Fourthly a release build is run. The settings are identical to the debug build except the release configuration says Release. Fifthly, copy the test files to a test folder. This was necessary in order to be able to run the Specflow tests later on from a known location.
Sixthly, create the BTDF msi package.
Seventhly, package the BDTF as a msi.
Eighthly, push the msi to Octopus.
Finally, publish the artifacts to use these in subsequent release pipelines.
The next thing to do is to use all these artifact in release pipelines.
Devops – create a release pipeline
Create a release with four jobs. For example see below.
The first step is to configure release pipeline to pick up the artifacts from the build pipeline.
Secondly the Dev -Create Octo Release job contains two tasks.
The first task runs a powershell script to terminate any instances that will stop the subsequent Octopus release uninstalling the target application. The second task “create Octopus release” and deploys it the Dev/Build BizTalk server. It is notable that an additional argument is necessary to avoid time outs during the octopus deployment.
Thirdly, the Run Specflow jobs runs if the Dev -Create Octo Release job is successful.
The Specflow task was difficult to setup because it would run all the tests without failure but then throw a run time error. The error was DTAExecutionHost.exe’ failed. After much anguish, the error disappears using a known test artifact folder.
Finally, the UAT – Deploy Octopus and Production – Deploy Octopus jobs are only run manually. They contain one task and only differ in the configured environment.
Conclusions
To conclude with I have shown how to use Azure Devops to build on-premise BizTalk 2016 Server applications. My task group can be downloaded as a quick start from here.
