Bringing things together so they stick

Interactions are hard to create. And harder to control.

When you are building a tech based offering, you will generally focus on building out the customer facing part or the innovative part of your business directly with your small team. But the innovative part of your offering can’t exist in a vacuum. It will need to inform and drive a variety of business functions from marketing, supply chain, support and finance.

As much as you might wish, you can’t build everything on your own. In fact, you shouldn’t build everything on your own. There are great enterprise software available for all business processes and this keeps increasing every year. Many of these will work better than what you can build yourselves due to their focus on solving specific problems. Further, buying such off-the-shelf solutions often works out much cheaper and faster than building yourselves. We are also in a golden period of enterprise SaaS and API software which are highly configurable and built for integrations.

The most common scenarios where products need to work together are:

• Integration with a CRM for lead management, customer support and complaints management
• Sending communications like SMS, Emails, WhatsApp messages
• Data warehouse for reporting and analytics

Unfortunately, the process of bringing two products together is usually strewn with timeline delays, budget overshoots, compromises to the promised end state and a lot of heartburn for the teams involved.

Product integrations are necessary, important and messy. Part of the messiness comes from the inherent nature of bringing together two different objects built separately. There is bound to be some friction. But I believe most of the messiness comes from the lack of appreciation of the complexity involved and not building out the necessary safeguards and plumbing to deal with issues.

Before we get into the details, let’s talk about the types of product integrations. What matters most from the perspective of getting two things working together seamlessly is the degree of separation. So the two things that you bring together can be:

• Internal modules of same product
• Different internal products
• One internal product and the other a 3rd party solution
• Two 3rd party solutions

While some of this blog post is relevant for building or integrating internal software, it is more relevant whenever 3rd party solutions come into the picture (where the degree of separation is highest and therefore the level of control and possible changes are lowest).

So what can be done to improve the odds of delivering successful integrations. A few things can help.

Align all teams on the outcome and keep reinforcing it. It is easy for different product teams to get into their narrow zones particularly when different business divisions or organizations are involved. It is important to upfront establish a single customer-backward success metric for the integration to align all parties. While there definitely needs to be clarity in the responsibilities of each team, ambiguity is bound to arise in the middle of any complex integration and having alignment and buy-in on what constitutes overall success of the project is important to deal with such situations. While this is a no-brainer, it is surprising how often lack of alignment and ownership by the teams involved is the root cause of a failed integration effort.

Don’t trust any features that are not currently a part of any third party software. Have a clear idea on what’s currently available, ideally by testing it. Lots of companies oversell what they have and casually assure to include items you need but they don’t currently have to their roadmap but those don’t usually materialize. Do not plan based on such assurances. If you really need such features, ensure your contract or payment terms to the vendor are strictly tied to the delivery of such features.

Know the limits of both the systems and operate within the limits. Every software system will have some limits. For example, a CRM system might have limits on the number of support tickets that can be created in parallel or within a minute. Unfortunately, these limits are not always explicitly called out by vendors (sometimes, they are unaware of it as well). If you are building out a large scale consumer or enterprise business, it is highly likely that you will hit the system limits at some point. You need to know these limits to ensure the system is designed to deal with it. These limits can be across different performance dimensions such as throughput and data volume. For example, let’s say that only up to 100 support tickets can be created within a minute by the CRM. While this might be well within the average number of tickets you might get, your peak load can be much higher than this limit. You might then need to have some kind of a buffering system that ensures you push out tickets only at the rate that the CRM can take.

For time-sensitive responses, integrate with multiple systems for redundancy. For critical systems such as payment during checkouts or support chat where any performance delays are as good as the system not working, try and integrate with multiple similar software. You can then manage the traffic you send to the systems based on their actual performance levels. This is obviously harder and more expensive than integrating with a single system but can give you a lot more control and provide higher overall performance guarantees as your product becomes bigger.

Involve data security teams upfront. If you work in a fairly large organization, whenever you have integrations with 3rd party software, your data security / cyber-security teams will need to approve of the integration. They would typically check for things like transfer of confidential customer data outside the organization, security of the interface protocols, etc. and see if the best practices are followed. This can take time. Do not wait until you are close to taking the system live to start the necessary processes. These would be common teams taking care of security across the organization and it takes time to get their availability and they can’t be harried to provide a sign-off. Understand what they look for and ensure they are part of the integration requirements.

Set up automated reconciliations between the two systems for monitoring any discrepancies. However good the initial integration is, errors can and will start to creep up over time. This can be due to any number of reasons ranging from unexpected scenarios and system infrastructure issues. You will need to have a separate mechanism that can periodically check the data between the systems and validate if there are any gaps and send out some kind of automated alerts or reports which can then be investigated manually. The frequency of these checks should be a function of the criticality of the process.

Originally published at http://vivekrajasekaran.com on November 13, 2021.