Data Integration vs Application Integration — The Yin & Yang of Your Digital Transformation

In today’s competitive IT landscape, it’s critical to automate business workflows. There are many technologies for process automation, including application integration and data integration. This will increase team productivity, improve customer experience and speed up decision making.
For example, retailers are building customized marketing campaigns for both in-store and virtual purchases and forecasting demand to optimize inventory levels. But to be successful, the retailer’s technology must be integrated with its manufacturer’s marketplaces. This is so it can automatically receive orders that flow straight through to billing, fulfillment and shipping. If not, a lot of time and resources are wasted in every step of the process. If the manufacturer has its order data and applications integrated with its customers’, the orders will automatically show up, be managed and be accelerated.

That’s the business side of things. Now, let’s discuss analytics. Businesses want to be able to analyze data. In the above example, for a manufacturer, this would entail the number of orders received within any given timeframe, its trend, order volume by customer, fulfillment SLA attained, etc. If business analysts or management want to inspect order volume, its longitudinal behavior, etc., they need to have easy and quick access through self-service analytics.

As a business process moves forward, control must be passed from one application to another. Applications need to be integrated, or, connected to each other to allow a process to move forward across applications.

Now imagine that the above manufacturer starts processing orders 24 hours after reception. The order request contains basic information, and the complete order data is loaded into the data warehouse using periodic batch jobs. In this case, data integration is used to extract the data from the order system and integrate it with the rest of the business systems. This includes systems for customers, products, and sales analytics, as well as for supply chain planning.

As you can see, organizations rely on both application and data integration to automate processes and share real-time and batch data. These are two distinct but essential capabilities: the yin and yang that businesses need to achieve digital transformation.

Let’s start with application integration, the yin of digital transformation. Application integration is used to connect different applications and is performed in (near) real-time to complete a business process or real-time data integration. This can be receiving a customer order. Another example is providing immediate customer data to a customer service representative for credit approval.

It’s helpful to understand API management in the context of integration: APIs have evolved as one of the most popular ways to integrate applications. This is because APIs are light, loosely coupled and can be secured and controlled. API management provides the ability to publish, subscribe, secure, monitor and manage APIs. API management doesn’t integrate applications. Rather, it’s an extension of application integration because app integration technologies provide the ability to create and author APIs. API management provides a means to control them.

Back to integration: Even ten years ago, enterprises would use enterprise service buses (ESBs), integration servers and message brokers to integrate their applications. However, integration platform as a service (iPaaS) has become the de facto technology for application integration. It quickly evolved to include data integration for multiple clouds and on-premises deployments.

The most basic use case of iPaaS technology is integrating customers’ on-premises SAP implementation with their cloud-based Salesforce CRM. In addition, iPaaS technology should:

• Support advanced data management needs
• Provide out-of-the-box connectivity options to popular cloud and on-premises systems and data stores
• Offer authoring tools to define a process, an API, or a data mapping easy enough to be used by developers, citizen integrators and analysts

Typical iPaaS technology also comes with developer and operations collaboration tools for version control, CICD and DevOps.

The yang of digital transformation is data integration. This is used to load or replicate new data from heterogenous data sources into a central repository such as a data warehouse. This provides a unified view of the business for analytics and reporting. Analytics includes dashboards, data mining, predictive analytics and self-service analytics.

Data integration is also used to migrate data from legacy systems to modern environments. Another example of data integration is consolidating data from operational systems into a data warehouse to reduce the impact on the operational systems. Data integration tools are extract, transform and load (ETL) tools. ETL tools are used to:

• Extract or ingest data from a variety of sources
• Provide data transformation capabilities to standardize or cleanse the data
• Load the data into the target system(s)

The ETL tools include a visual designer to help developers create data pipeline mappings to extract, transform and load the data into the data warehouse.

Application integration is used for process or API-centric integration. Data integration is used for data-centric integration via batch jobs that are processed periodically — weekly, daily, hourly or as needed. This data-centric approach is designed for collecting data for historical analysis. It consists of integrating millions of sales transactions, orders, insurance claims, clinical tracking activities, machine productions and other types of data.

With digital transformation, more and more data integration solutions provide the capability to integrate streaming and high-volume, real-time data to process time-sensitive data. This includes sensor data to avoid production disruptions, store transactions to prevent fraud while it is happening and supply chain routing to avoid weather delays. The majority of the analytics solutions have been migrated to the cloud with cloud data warehouse solutions but still rely on data integration as the key technology to manage their data.

Hopefully, we have made it clear that to achieve digital transformation you need both data and application integration. Moreover, to be efficient and cost effective, organizations want these tools to be part of the same solution.

However, having both data and application integration capabilities in one unified solution is just the beginning. Data management is critical because what good is data if it can’t be trusted or secured? It’s not very useful if you can’t consolidate the data to provide a comprehensive view of your customer, is it? And then there is artificial intelligence (AI), which harnesses the power of your data and metadata to automate your processes, making these more efficient and intelligent.

Focusing on business continuity is important in today’s climate, which calls for “lights-out” operations. But think about it. If you didn’t have your data and applications connected, would you be able to achieve continuous operations?

Going back to the example above: Unless you integrate your applications, you will be manually moving your business process from one system to another, perhaps by email notification. We have all heard stories of manual data sharing through spreadsheets. This makes the process slow, unreliable and insecure. Now think about a business process “auto-correcting” for an order that halted due to an apparent lack of inventory (an error), even with substantial inventory at-hand — the result of application integration and data integration running on the same platform.

How? A job running on your data integration service failed, resulting in incorrect inventory data. Since the application integration service is running on the same platform, it has immediate knowledge of this failure. It waits for the inventory update to complete successfully and in the meantime even sends an email to the customer informing them of a delay in their order fulfillment.

Now think about the result if these two services operated on independent platforms. This would perhaps require all kinds of manual intervention and correcting for errors. This could cause delays and, in the worst case, a loss of revenue. Do you still think it’s a good idea for application and data integration to work independently?

• Five things to look for in an iPaaS.
• What is data integration?
• Learn to effectively integrate data and applications with iPaaS.
• AI-powered iPaaS automates business processes with TDWI’s Insight Accelerator.
• Learn more about Informatica Cloud Application Integration and Cloud Data Integration, services of the Informatica Intelligent Data Management Cloud™ (IDMC).
• Experience a 30-day free trial of Informatica iPaaS, a service of IDMC.
• Free Data Integration Software