Enterprise integration

Large organizations have a large number of applications running in separate silos that need to share data and functionality in order to operate in a unified and consistent way. The process of linking such applications within a single organization, to enable sharing of data and business processes, is called enterprise application integration (EAI).

Similarly, organizations also need to share data and functionality in a controlled way among themselves. They need to integrate and automate the key business processes that extend outside the walls of the organizations. The latter is an extension of EAI and achieved by exchanging structured messages using agreed upon message standards referred to as business-to-business (B2B) integration.

Fundamentally, both terms refer to the process of integrating data and functionality that spans across multiple systems and sometimes parties. The systems and business processes in these organizations are evolving, and so is the technology enabling B2B unification.

Evolution of integration

There isn’t a year when certain integration technologies became mainstream; they gradually evolved and built on top of each other. Rather than focusing on the specific technology and year, let’s try to observe the progression that happened over the decades and see why blockchain is the next technology iteration.

Evolution of integration technologies

Next we will explore briefly the main technological advances in each evolutionary step listed in the table above.

Data integration

This is one of the oldest mechanisms for information access across different systems with the following two primary examples:

• Common database approach is used for system integration within organizations.
• File sharing method is used for within and cross-organization data exchange. With universal protocols such as FTP, file sharing allows exchange of application data running across machines and operating systems.

But both approaches are non-real-time, batch-based integrations with limitations around scalability and reliability.

Functionality integration

While data integration provided non-real-time data exchange, the methods described here allow real-time data and importantly functionality exchange:

• Remote procedure call provides significant improvements over low-level socket-based integration by hiding networking and data marshaling complexity. But it is an early generation, language-dependent, point-to-point, client-server architecture.
• Object request broker architecture (with CORBA, DCOM, RMI implementations) introduces the broker component, which allows multiple applications in different languages to reuse the same infrastructure and talk to each other in a peer-to-peer fashion. In addition, the CORBA model has the notion of naming, security, concurrency, transactionality, registry and language-independent interface definition.
• Messaging introduces temporal decoupling between applications and ensures guaranteed asynchronous message delivery.

So far we have seen many technology improvements, but they are primarily focused on system integration rather than application integration aspects. From batch to real-time data exchange, from point-to-point to peer-to-peer, from synchronous to asynchronous, these methods do not care or control what is the type of data they exchange, nor force or validate it. Still, this early generation integration infrastructure enabled B2B integrations by exchanging EDI-formatted data for example, but without any understanding of the data, nor the business process, it is part of.

With CORBA, we have early attempts of interface definitions, and services that are useful for application integration.

Service-oriented architecture

The main aspects of SOA that are relevant for our purpose are Web Services standards. XML providing language-independent format for exchange of data, SOAP providing common message format and WSDL providing an independent format for describing service interfaces, form the foundation of web services. These standards, combined with ESB and BPM implementations, made integrations focus on the business integration semantics, whereas the prior technologies were enabling system integration primarily.

Web services allowed systems not to exchange data blindly, but to have machine readable contracts and interface definitions. Such contracts would allow a system to understand and validate the data (up to a degree) before interacting with the other system.

I also include microservices architectural style here, as in its core, it builds and improves over SOA and ESBs. The primary evolution during this phase is around distributed system decomposition and transition from WS to REST-based interaction.

In summary, this is the phase where, on top of common protocols, distributed systems also got common standards and contracts definitions.

Blockchain-based integration

While exchanging data over common protocols and standards helps, the service contracts do not provide insight about the business processes hidden behind the contracts and running on remote systems. A request might be valid according to the contract, but invalid depending on the business processes’ current state. That is even more problematic when integration is not between two parties, as in the client-server model, but among multiple equally involved parties in a peer-to-peer model.

Sometimes multiple parties are part of the same business process, which is owned by no one party but all parties. A prerequisite for a proper functioning of such a multi-party interaction is transparency of the common business process and its current state. All that makes the blockchain technology very attractive for implementing distributed business processes among multiple parties.

This model extends the use of shared protocols and service contracts with shared business processes and contained state. With blockchain, all participating entities share the same business process in the form of smart contracts. But in order to validate the requests, process and come to the same conclusion, the business processes need also the same state, and that is achieved through the distributed ledger. Sharing all the past states of a smart contract is not a goal by itself, but a prerequisite of the shared business process runtime.

Looked at from this angle, blockchain can be viewed as the next step in the integration evolution. As we will see below, blockchain networks act as a kind of distributed ESB and BPM machinery that are not contained within a single business entity, but spanning multiple organizations.

Integration technology moving into the space between systems

First the protocols (such as FTP), then the API contracts (WSDL, SOAP) and now the business processes themselves (smart contracts) and their data are moving outside of the organizations, into the common shared space, and become part of the integration infrastructure. In some respect, this trend is analogous to how cross-cutting responsibilities of microservices are moving from within services into the supporting platforms.

With blockchain, common data models and now business processes are moving out of the organizations into the shared business networks. Something to note is that this move is not universally applicable and it is not likely to become a mainstream integration mechanism. Such a move is only possible when all participants in the network have the same understanding of data models and business processes; hence, it is applicable only in certain industries where the processes can be standardized, such as finance, supply chain, health care, etc.

Generations of integrations

Having done some chronological technology progression follow-up, let’s have a more broad look at the B2B integration evolution and its main stages.

First generation: system integration protocols

This is the generation of integration technology before CORBA and SOA, enabling mainly data exchange over common protocols but without an understanding of the data, contracts and business processes:

• Integration model: client-server, where the server component is controlled by one party only; examples are databases, file servers, message brokers, etc.
• Explicit, shared infrastructure: low-level system protocols and APIs such as FTP.
• Implicit, not shared infrastructure: application contracts, data formats, business processes not part of the common integration infrastructure.

Second generation: application integration contracts

This generation of integration technology uses the system protocols from previous years and allows applications to share their APIs in the form of universal contracts. This is the next level of integration, where both applications understand the data, its structure, possible error conditions, but not the business process and current state behind it in the other systems:

• Integration model: client-server model with APIs described by contracts.
• Explicit, shared infrastructure: protocols, application contracts, and API definitions.
• Implicit, not shared infrastructure: business processes and remote state are still private.

Third generation: distributed business processes

The blockchain-based generation, which still has to prove itself as a viable enterprise architecture, goes a step further. It uses peer-to-peer protocols, and shares business processes with state across multiple systems that are controlled by parties not trusting each other. While previous integration generations required shared understanding of protocol or APIs, this relies on common understanding of the full business process and its current state. Only then it makes sense and pays off to form a cross-organization distributed business process network:

• Integration model: multi-party, peer-to-peer integration, by forming business networks with distributed business processes.
• Explicit, shared infrastructure: business process and its required state.
• Implicit, not shared infrastructure: other non-process related state.

There are many blockchain-based projects that are taking different approaches for solving the business integration challenges. In no particular, order here are some of the most popular and interesting permissioned open-source blockchain projects targeting the B2B integration space:

• Hyperledger Fabric is one of the most popular and advanced blockchain frameworks, initially developed by IBM, and now part of Linux Foundation.
• Hyperledger Sawtooth is another Linux Foundation distributed project developed initially by Intel. It is popular for its modularity and full component replaceability.
• Quorum is an enterprise-focused distribution of Ethereum.
• Corda is another project that builds on top of existing JVM-based middleware technologies and enables organizations to transact with contracts and exchange value.

There are already many business networks built with the above projects, enabling network member organizations to integrate and interact with each other using this new integration model.

In addition to these full-stack blockchain projects that provide network nodes, there also are hybrid approaches. For example, Unibright is a project that aims to connect internal business processes defined in familiar standards such as BPMN with existing blockchain networks by automatically generating smart contracts. The smart contracts can be generated for public or private blockchains, which can act as another integration pillar among organizations.

Recently, there are many blockchain experiments in many fields of life. While public blockchains generate all the hype by promising to change the world, private and permissioned blockchains are promising less, but are advancing steadily.

Conclusion

Enterprise integration has multiple nuances. Integration challenges within an organization, where all systems are controlled by one entity and participants have some degree of trust to each other, are mostly addressed by modern ESBs, BPMs and microservices architectures. But when it comes to multi-party B2B integration, there are additional challenges. These systems are controlled by multiple organizations, have no visibility of the business processes and do not trust each other. In these scenarios, we see organizations experimenting with a new breed of blockchain-based technology that relies not only on sharing of the protocols and contracts but sharing of the end-to-end business processes and state.

And this trend is aligned with the general direction integration has been evolving over the years: from sharing the very minimum protocols, to sharing and exposing more and more in the form of contracts, APIs and now business processes.

This shared integration infrastructure enables new transparent integration models where the previously private business processes are now jointly owned, agreed, built, maintained and standardized using the open-source collaboration model. This can motivate organizations to share business processes and form networks to benefit further from joint innovation, standardization and deeper integration in general.