Event Driven Architecture Patterns: Event Normalizer
As a (hopefully) regular series on this blog, I will seek to provide a pattern library of patterns found and used extensively within an Event Driven Architecture. I’d love to get feedback from other practitioners on these patterns.
The first pattern I will document is the Event Normalizer.
Title
Event Normalizer
Description
Events external to the event driven architecture – be they sourced from external Message Emitters (such as a trading exchange or market data feed), or enterprise internal Message Emitters (such as a legacy system) – need to be normalized before they can be consumed within the architecture.
This normalization process may make structural changes to the event’s data format (e.g. format conversion) and add particular enrichments to a event (e.g. add geographic attributes). Where additional attributes are added, they would often be sourced from a master data engine.
Diagram
Benefits
- Reduces coupling between Event Sinks or Event Processors and the Event Emitter. This has a whole host of benefits, particularly allowing an implicit styled architecture (in other words, emitters have no explicit knowledge of sinks or processors) which fosters dynamic determinism and extreme loose coupling
- Disparate event attributes become mastered into the organizational golden definition. For example, using the diagram elements above, lets imagine that within Format A, a counterparty – Goldman Sachs – is referred to as “GS”, whilst in Format C, it is referred to as “231238″. Internal Format B may already make use of the internal master definition of the counterparty. The Event Normalizer then passes the original event source’s counterparty information on to a Master Data engine, and the mastered counterparty returned is the correct internal representation. Any additional processing taking place down stream just needs to be able to interpret the internal representation – it has no need to understand the original representation.
Limitations
- The Event Normalizer is a potential performance bottleneck
- The Normalized Event needs typing and a defined machine definition (such as a well defined XML Schema)
- The Event Normalizer is also a bottleneck to the dynamic addition of new event sources (although this depends on the design of the master/reference data engine)
- The organization should typically have sophisticated processes already in place for managing master/reference data
Recommended Usage
The Event Normalizer is suitable for use in the following scenarios:
- The variety of event classifications (e.g. a trade event, a market data event) is less than the the number of Event Emitter formats. For example, a trading organization that has interest in processing trade events from a large number of different exchanges and brokers
- A high speed, well maintained, master/reference data engine is available within the organization
Related Patterns
- The Normalizer pattern within the Enterprise Application Integration library, although this is more focused on format detection and standardization
Pattern Documentation History
- Version 1.0 – Created
- Version 1.1 – Clarifications around the difference between the EAI Normalizer and this pattern. Also, an improved diagram.
Most CEP patterns seems to be variations of EAI patterns. Found, for example, here http://www.eaipatterns.com/
One guys message is another ones event. It might not be clear what the difference between EAI (message processing) and CEP is…
So these pattern could possible be used used in event processing too. Some re-naming and maybe little changes on semantics they might be CEP patterns too. Perhaps depending on your view what an event is, some might not make sense. Like changing the contents of an event if you see events as something that are created once and then never changed.
You’re absolutely correct in the relationship between the two – the Event Normalizer pattern is directly related to the EAI Normalizer pattern. There is however an implied difference that is not that clear in my original post – the enrichment of the event with additional data. I will expand on the pattern to make this clearer.
On another note – beware the CEP = EDA evaluation. CEP is a processing style within EDA, not the same thing. EDAs may also run with simple event processing and stream event processing.
Whoops. I was indeed careless with the TLAs here. CEP and EDA is not the same thing.