Last Orchestration

Today I wrote my last orchestration and left Datacom after ten years. This blog remembers the good and bad times through the eyes of this blog.

Before Blogging – 1st orchestration

I joined Datacom in January 2007 as a systems consultant. My first project was to stabilise a greenfields implementation of BizTalk 2006. Greg Forsythe designed this for BizTalk 2004 but Datacom migrated this to BizTalk 2006 before go live. Ratnakar Garikipati, Capree Lim, Coen (Colin) Dijkgraaf, Kiril Petrov and Francis Yuen where the team members. There was a wide range of BizTalk artefact’s namely;

Party resolution to dynamically route messages to different ports.
POP3 implementation with a flat file conversion.
A dynamic FTP and SMTP implementation interface to over 60 branches.
Interfacing with the Oracle financials XML gateway with HIWG invoices.
A dynamic FTP interface to over 60 branches.

Eventually I did write about my experiences in two of my blogs;

After Thiago Almeida – 2nd Orchestration

Thiago Almeida joined Datacom three months after I did. I knew him previously as a “kid in short pants straight of the boat from Brazil” who worked with me at the Auckland District Health Board several years earlier. This was in the days before BizTalk 2002 and we did our HL7 messaging using an old product called eGate which is now almost extinct. This began a long friendship and I learnt a lot from Thiago while he was at Datacom.

I blame Thiago for introducing me to BizTalk in the first place. One day he came to me and said “I have migrated some BizTalk 2002 solutions to BizTalk 2004 and I am going to hand this over to you to support….and by the way I am leaving ADHB next week.”. This was the first of many curveballs that Thiago would throw me over the years. The worst part of the solution was a custom Atomic file adapter that processed files from a UNIX NTFS file share. The host that it was on had to be restarted every 14 days because of a memory leak. Needless to say I very quickly learnt some BizTalk skills and had become reasonably competent before I moved on to Datacom.

Early years – 3rd Orchestration

Thiago and I became very exited about the new WCF adapters in BizTalk 2006R2. This became a dominant theme in most of the solutions that we created. It is amazing to see how WCF has become less relevant in middleware solutions these days.

It took me a while to get to use WCF adapters in anger because I was stuck on BizTalk 2006 projects. Ben Gracewood started me on a long stream of work that went on and off for 3 years. The BizTalk 2006 server connects a AS400 system to SAP and other systems. One of the most interesting projects I did during that time was integrating to a voice activated picking system. I still remember that the voice system had me laughing in the aisles because it pronounced all the NZ Maori place names in American English. Māngere became Man-Gerry.

Some blog posts during this period were;

An alternative to using the POP3 adapter to read email attachments describes how I got around a ban on using the POP3 protocol.
DB2 BizTalk Adapters: Executing a program on an AS400 using a stored procedure outlines a way to retrieve data from a COBOL RPG program. This was the first of many integration patterns taught to me Chris Scott while he was at Datacom.
Disassembling Excel files shows how to use the custom Farpoint adapter to parse the file. This was the first of 3 blogs that writes about consuming Excel files in BizTalk. See FarPoint Spread for BizTalk 2010 and https://exceldatareader.codeplex.com/.

Curveballs – 4th Orchestration

Finally my chance came to use the new WCF adapters. Thiago came me and said “I’m going on holiday to Brazil, while I am away I want you and Chris Petit to migrate a BizTalk 2004 server to a BizTalk 2006R2 server”. This second curveball throws me a challenge that I readily accepted. This involved converting all the 2004 SOAP adapter interfaces to WCF adapters. Upgrading the Web Services Enhancement adapters (WSE) adapters from version 1.0 to 2.0 made me wonder who created this abomination. Thank God this adapter is now deprecated.

Subsequently Chris and I worked together on a SSIS solution to sort parcels by street address.

I did not post anything about these experiences.

Going Solo – 4th Orchestration

Beginning in 2009 I worked solo on a customer site for the entire year. Thiago started the famous Datacom monthly BizTalk breakfast during this time, at my suggestion because he wanted to catch up with what I was doing.

I designed and implemented a BizTalk 2006 R2 exchange that connects a MOSS Online Sales site with several legacy backend systems. Using WCF protocols for all service endpoints, SQL connections and MSMQ messaging secured a low latency solution.

Some posts that originating from this time were;

Performance of the WCF-Basic HTTP Adapter and Using LoadGen to test Wcf-NetTCP binding illustrates how to measure latency of BizTalk solutions.
How to copy the entire node to element of string type in a map is my most highly viewed post of all time.
Failed to update MessageBox server while creating BizTalk Host

Award Winner – 6th Orchestration

I was given a “Greenfields” customer to replace eGate with BizTalk server 2010. The final solution integrates vending machines, car park barriers and door controls with a gaming system. A long running orchestration maintains the state of a punter during their journey through a casino. This was the most challenging BizTalk implementation that I have done because I had to ensure that all request response latency where less than 1 second 99% of the time. Along the way I meet Shadab Anwer who then joined Datacom for awhile. I was pleased to win a partner of the year award for this work and was promoted to Senior Systems Consultant at Datacom.

Some posts for this assignment where;

Connecting BizTalk 2010 to a GVC-LC Vending Machine
The GVCLC Adapter-My First WCFLOB Adapter. This was the first custom WCF LOB adapter I ever wrote. I acknowledge Chris Scott for teaching me the how sockets work under the hood. Thiago introduced me here to Bill Chesnut for the first time. Bill gave me invaluable advice on how to build a first class custom BizTalk adapter. Once Thiago left Datacom, Bill became the only source of advice that I could enlist if I got stuck.
GVCLC Custom WCF Adapter-Part 2
Does the call rules shape create an orchestration persistence point

A Dead BizTalk Server – 7th Orchestration

Thiago was sitting at his desk and got a call that said “Our BizTalk Server has flat-lined, can you help us?”. This was the beginning of a long relationship with a new customer that continues to this day. Sure enough when we rocked up the BizTalk server was sitting at 100% CPU and the SQL server was not much better. The event log was a train wreck. We managed to stabilise the server and reduce the CPU with a few tweaks. Furthermore we identified that there was a lot of bad code running on this server. Our brief became keeping the BizTalk 2006R2 and 2002 Server going until it could be replaced. Unfortunately we did too good a job and within one year we where being asked to upgrade to BizTalk Server 2o10.

Finally in 2015, I lead the largest integration project that Datacom Auckland have every done with this same customer; integrating Demandware, Salesforce, Oracle and many legacy systems for this customer. I talked about it in a Integration Monday session here. This was one of the most satisfying engagements I had in all my time at Datacom. This is a tremendous example of how partnering with company and sticking with them through tough times will eventually win great returns. This is where I first worked with Merry Hee. It is shame that I am just leaving Datacom as Merry has joined.

Some posts from this journey are;

Post Thiago – 8th Orchestration

All good things come to an end. Thiago left Datacom in 2011 . I feel very privileged to have worked with Thiago and I would not be the developer that I am today, without the time I spent under his leadership. For a time after he left I was all at sea until I realised that I had to pick up from where he had left off.

Assuming the role of technical leadership of the Datacom integration team I was given the grandiose title Principle Integration Architect. Firstly, I established a semi-regular Friday Framework session at Datacom talking about integration topics that still continues to this day. Secondly, I began to encourage the team to move to continuous integration. Unit testing became mandatory as we started to introduce build servers in to all our projects. Thirdly, I started a regular Friday stand up where the entire integration team shared what they were working on. Fourthly I wrote or improved several tools and frameworks to help the Datacom team.

Some of the tools were;

An email processing framework that routed random attachments using the Business Rules Engine.
A convertEXceltoCSV pipeline component.
An improved version of the WCF Loopback adapter.
An improved version of the ESB Exception Management Portal.
An organized BizTalk solution structure for Visual studio 2010 and 2013. This is based on the example in Dan Rosanova’s book.
The BizTalk Documenter 2013 with Colin Dijkgraaf.

The Darkness – 9th Orchestration

And so 2012 heralds the worst time of my working life. Three projects became death marches and ended badly.

The first project replaces a monolithic COBOL based application with modern applications. The solution uses BizTalk and my task was to implement a legacy adapter or façade pattern. This all ended in tears because the new application was turned off because it had issues. The BizTalk server solution remained in place but all it did was replace an aging Cold Fusion server and introduce more complexity into the solution.

The second project involves using BizTalk Server, SharePoint and Microsoft CRM to create a document management system. I arrived back from holiday and was asked to “..get my butt down the road and get that mess sorted…”. This was a “square peg in a round hole”. After many weeks with the help of two other very talented BizTalk developers(Colin Dijkgraaf and Mahindra Morar) we had something that just worked.

The third project was a major refresh of an exchange. The choice of technology here was good but the customer expectations did not match with what we produced. It all turned into a contest of wills and we decided to part ways.

It is credit to all the Datacom people(Chris Scott, Shabad Anwer, Ian Hui, Johann Cooper, Ottolina Pentz, Shikhar Baigent, Cherie King, David Trimble, Colin Dijkgraaf and Mahindra Morar) that I worked with that we eventually got to the end of these projects but it took a large toll on me. At the same time my daughter became incredibly sick and I took some time off. It seemed as if nothing was going my way.

My posts from this time are surprisingly cheerful;

The Light – 10th Orchestration

At this point I sat down and set some goals to prove that I still had my BizTalk macho. Firstly to lead a big successful project. Secondly to demonstrate technical credibility with the integration community.

I achieved my first goal when I lead the biggest integration project that Datacom has done and I referred to this in the 7th orchestration. Looking back this was start of a ramp up in the scale of projects being done by all of my colleagues.

I think have meet the second goal in many ways;

I spoke at BizTalk Saturday run by Bill Chesnut.
BizTalk Documenter Codeplex project was refreshed with my Colin Dijkgraaf. See above for a link.
Publishing “SOA Patterns with BizTalk Server 2013 and Microsoft Azure – Second Edition” by Mark Brimble, Johann Cooper, Coen Dijkgraaf and Mahindra Morar. This was a once in lifetime experiencing writing a book with three of my best friends.
Organized and spoke at Integration Saturday 2015.
Integration Monday – What’s wrong and right with the BizTalk WCF-WebHTTP adapter
Ignite NZ 2016 – What’s New in BizTalk 2016
Integration Monday – BizTalk Server with SQL Availability On Databases
Organized Auckland Global Integration Bootcamp 2017.

Last month I turned 60 and decided that I wanted to spend more time doing the things I enjoy outside of work. On reflection doing another large project or doing my first project with the new azure technologies is not the be all and end of all of life.

I will miss many people at Datacom and will watch Datacom progress into a new era knowing that I once had the privilege of working there. There are two people I wish to single out who made a big difference to my journey. There are many others and I apologise for not mentioning you. I will especially miss my technical chats and Friday lunches with Mahindra Morar. He is the best technical mind I have ever known. I will also miss my daily slew of emails and chats with my favourite project manager Cherie King . Her skill at organizing me and others towards a common goal continues to amaze me.

Everything ends and that is always sad but there is always new beginnings and that is happy. I am looking forward to a new challenge and a new beginning. Be Happy.

Last Orchestration – Time Everyone