Event Driven Architecture and Cloud Computing
Although the primary benefits of cloud computing at this stage appear to be largely accounting related (i.e. moving capital expenses to operational expenses), there may well be some additional benefits coming down the line as vendors and IT teams get a handle on exactly what it is and how best to leverage it. EDAs which make use of fully self contained messages, and appropriate SOA standards, are well placed to make the move from data center to cloud – if the kind of processing you’re looking like moving to the cloud is inline with what clouds are best suitable for today.
The complexity with mixing EDA and Cloud Computing today is that the kind of algorithms and processing styles best suited to cloud computing are in effect, largely the opposite of the problem EDA tries to solve. EDA is all about near instant access to business events, where ever its needed and by everyone who needs it. Cloud computing appears more suitable for large scale batch oriented processing, using techniques such as MapReduce to manage the processing of huge data sets in parallel. For example, it is highly unlikely a large scale realtime P&L operation for a trading firm could be run on a cloud today.
This doesn’t mean EDA has no possible use near a cloud though: batch CEP type processing can be run in these environments – for example, the Apache Mahout project aims to implement a good baseline of machine learning algorithms on top of a MapReduce platform, and with a large number of collected events, this may prove useful.
Some additional things to consider about cloud computing that not all vendors like to be clear about:
- How do you manage the data geographically (for example, the data protection laws in Europe make things complicated)?
- How do you move the data around externally to the firm with sufficient value to be properly processed, yet still secure enough to appease the auditors? You need to weigh up the rewards & risks of using a cheaper large scale multi-tenant operation such as Amazon over running a private cloud using suppliers such as IBM.
- How do you deal with failure? For example, a remote cloud center may be taken out from the corporate point of view if some Trawler broke an undersea cable. What happens if your large batch operations are unable to be executed for a few days?
- How do you deal with a cloud operator disappearing or closing down? There are little to no standards shared between service providers.
- The complexity of network infrastructure management and software monitoring jumps dramatically – how will this be dealt with?
Incidentally, I have bumped into a few people using cloud computing in a financial services environment (who also have various forms of EDA), and they tend to be only within two contexts: backup and compliance.
From my experiance I see that you need more work than you could imagine when building something that will live naturally in a Cloud. One thing that helps is to build everything in an event driven manner. We have largely been saved by the fact that the product I’m working on is totally event driven, even internally. This does not give optimal performance on a single server. But when the world consists of a huge amount of cheap servers this is really nice.
For the same reason my best guess is that it will take some (long) time before a large part of the worlds server apps will execute in a Cloud smoothly. They are not built as building blocks for attachement into an event driven architecture. But gradually I start to see this idea of using an event driven approach even internally inside applications gaining ground.
One nice benefit is that you don’t have to change paradigm when going from the outside (if a part of an event driven architecture) to the inside of the system. Everything sort of looks the same on the macro and micro levels.
Hi again
You’re correct in that the move to cloud can be more complex than expected – in fact, my development teams are stumped as to how to solve the issue of large scale realtime processing within a commodity cloud environment. As a result, we’re in the process of connecting up with some researchers specialized in this subject to see if we can begin a shared research project. I will blog any outcomes on this, although it is very much still early days