This blog post explains the importance of using patterns in integration architecture and provides guidance on how to succeed. It covers the four steps of integration design, along with pattern examples such as Pipes-and-Filters, Canonical, and Enrich. The post also discusses the VETRO responsibility and how to implement it. Overall, this blog post is a useful resource for anyone looking to improve their integration architecture skills.
One of the requirements in the Certified Integrator course is about patterns:
“An integration shall be designed using the industry standard model with system, service, contract and integration parts. This shall be done technology neutral with identification and dependencies in the integration.”
Let’s dig deeper into this very essential part of integration architecture.
What are patterns and why are they important?
When designing an integration solution, it is important to use patterns that provide technology-independent design guidance. This will make it easier to interpret documentations of existing solutions and explain the solution to others.
Having your patterns figured out will make it easier to scope new projects and will facilitate better planning and shorten time-to-market.
The steps of integration design
Model you integration by following four simple steps when applying the concept model:
- Identify your systems
- Identify services provided by systems
- Identify contracts (i.e. consumed services)
- Identify e2e integration (i.e. one or more contracts)
Then, clearly state dependencies in the conceptual design, such as:
- “A system provides a service”
- “A system consumes a service through a contract”
- “An integration consists of one or more contracts”
Pattern examples
When discussing patterns in integrations, it's essential to start with the book "Enterprise Integration Patterns" (EIP), which contains 65 well-defined patterns. If you haven't read it yet, I highly recommend doing so!
One of the most well-known patterns in the book is the Pipes-and-Filters pattern, which involves using a filter to act on a message and a pipe to transport it. This pattern is utilized in the Certified Integrator methodology, where a filter is defined as a "System with a Service that consumes messages through a contract," and the pipe can employ techniques such as JMS, Kafka, HTTP, and more.
Now, let’s look into how to design a basic flow that involves two source systems and one target system. We assume that we know how System A and System B will send the data, and that System C will receive the data through the contracts.
To start off, we can use a Pipes-and-Filters pattern along with a three-layer architecture.
The three-layer architecture consists of three layers: System layer, Process layer, and Experience layer. Each layer contains a set of services and components, and communication between layers is done using well-defined protocols.
In the basic flow, we can place one filter in each layer, except for the system layer where we have one filter per system. The overall flow would look like this: "System --> Process --> Experience --> System".
The Canonical Pattern
With this approach, we can implement the canonical pattern from EIP, which should be located between the Process and Experience layers. This approach assigns the responsibility of transforming the target and source information models to and from the canonical format to the Process and Experience layers.
A practical way to build the canonical format is to begin at the bottom by defining names, objects, and their relationships. Also, it is important to limit the scope of the canonical format to a specific and well-known domain. Creating the canonical format from the top can cause it to become too complex, requiring a significant amount of effort to design.
The Enrich Pattern
Another pattern from EIP is Enrich. In our case, Enrich can be implemented in the Process Layer if the canonical format requires data from various sources to be included in one object. Alternatively, it can be implemented in the Experience Layer. Since our flow includes multiple objects in the canonical format, the Enrich pattern is best suited for the Experience Layer.
Another useful pattern involves validating the incoming and outgoing messages of every filter against the contract.
Fulfilling the VETRO Responsibility
VETRO stands for Validation, Enrich, Transform, Route, and Operate. I view them as the primary responsibilities of the integration layer. If the integration layer does more than this, it could be considered building an application instead.
- Validation involves validating request messages, which is fulfilled through checking the contracts for each filter.
- Enrich is fulfilled when we merge the two objects into one message.
- Transform involves converting messages from backend systems to API-friendly data representations such as JSON or XML. This is fulfilled through the transformation between different information models; A, B, C and canonical format.
We have almost every aspect of VETRO in our flow, with only Route and Operate left. Operate is typically handled in the system layer, where a safe and reliable API is provided. Basic filters and pipes are used to route messages to the correct filters. In our case, this is fairly straightforward.
In practice, it is common to implement the Validation, the Transformation, the Routing and the Operation part of VETRO. The Enrich is not always necessary.
The final result will look like this:
There’s only so much you can say in a blog post, and even if we only scratched the surface this time, I hope you gained some new insights or at least a spark to learn more. 🙂
Thanks for reading and don’t hesitate to contact me if you have any feedback or questions!
Links
Certified Integrator
A quality standard for system integrations, providing best-practice guidelines for more secure and reusable integrations.
certifiedintegrator.com/get-certified
Starlify
Build better integrations by collaboration and insight.
Starlify brings insight into your organisation’s system integration assets by collecting them all in one place. It will help you focus your efforts, and speed up integration delivery times.
