When Enterprise Architecture and BI collide

Last week I was in London at the Enterprise Architecture Conference Europe, first off I must say that this was a really enlightening conference, containing both good and bad examples of EA.  I'm a firm believer that those presenters who are brave enough to stand up and say "this just didn't work out" are often more valuable than those who want to represent everything as an unmitigated success.  For a while now I've been working around the idea of a post on how your EA strategy can and should impact   your BI/Big Data/Enterprise Analytics (delete as appropriate) strategy.

For those that have grown up on the Kimball method of building data warehouses, there might seem to be one solution of "we do masses of up front analysis and build a dimensional data warehouse", while this is a gross exageration it sums up many approaches I have seen.  The stock BI strategy has lots of nice, fairly obvious statements of "do away with data silos", "single version of the truth", "driven by business requirements" which are all very nice, but how do you actually go about achieving this? If you have the ideal of a CEO/CFO who is a strong advocate of the value of BI and is trusting enough to leave the specialists to get on with it, then you are truly blessed. But this is not always the case and there are often many other factors that now have to be taken into account.  With the velocity of change now increasing massively, not just in technical terms but business change we need to step back and look at the big picture for a moment.

Taking the view from the top, an enterprise or company is a collection  of people, processes and data working towards a common goal.  Our role as BI professionals is to help the enterprise do this more effectively, by enabling an increase in revenue, reducing costs or avoiding costs.  Enterprise Architecture is also a profession that aligns with this.  If you are not familiar with EA I'd recommend reading the excellent book "Enterprise Architecture as Strategy" by Jeanne W. Ross, Peter Weill and David C. Robertson.

There needs to be a clear differentiation between "Enterprise Architecture" and "IT Architecture for an Enterprise"  I've come across several IT architects who describe themselves as "Enterprise architects" when they only deal in IT.  Enterprise Architecture is the view of the business processes, their standardisation and integration and the underlying systems that support them.  Sometimes what was "Enterprise Architecture" is now being called "Business Architecture". A simple example of the difference is the IT view of "how can I redesign or implement this system to be more efficient?", the EA view is "how can I redesign or implement the processes supported by this system to be more efficient?".  There is no point in building the most perfect and cost effective IT system to support a process that no longer needs to be done.

While I'm on the topic one common theme I've come across is an odd behaviour of a commonly held belief that "IT cannot lead business strategy". While it is true to an extent that just because there "is" a new technology you should not necessarily use it, to be blinkered to new technical opportunities is just as bad.  It's much more helpful to see IT and the business as having a symbiotic relationship, each a separate entity with different drivers and opportunities but ultimately part of the same ecosystem, mutually dependent no the success of the other.

So back to the thread,  whether it is explicitly stated or not your Enterprise has an architecture, in my experience it mostly got there by accident rather than design.    The MIT Sloan Centre for information research  demonstrate 4 different operating models depending on the level of business process integration and business process standardisation.

By default most businesses subconsciously fall into the Diversification category, but this should not necessarily be seen as a bad thing, sometimes this is the most effective and cost efficient model for operating in certain markets.  Let me give my paraphrasing for each category, both the "formal" and reality of whats happening on the ground:

• Diversification:  

• Formally - Each business unit has the flexibility to implement its own services and products as it sees fit, there may be some shared services but generally each unit has the flexibility to define, build and run its own services and processes as required.  
• The reality - Its chaos out there.  Lots of duplication of systems, different processes for performing the same action, multiple business definitions for the same entity. Rapid pace of uncontrolled change.  You will probably also come across business units implementing their own IT systems. 

•  Coordination:
• Formally - A high degree of data integration, prebuilt delivery channels and standardised technology.  Each business unit still has a degree flexibility variation of the processes it uses on top of the standard services.
• The reality -  Lots of effort to define and enforce the standards.  The body responsible for enforcing the standards often seen as an obstacle to progress leading to a tendency to move back down to the diversification model.

•  Replication:
• Formally - Pretty much a franchise model, standard branding and processes reused repeatedly.  High efficiency from reduced risk of implementing new untried processes.  Local data ownership but enterprise wide definitions as part of the process.
•  The reality - replication of data systems and small local variations to standards.  No clear view of the customer gives a risk of competing against yourself for business.  Continual risk that the processes may be seen a limiting factor and an obstacle to progress leading to a tendency to move back to a diversification model.
• Unification:
• Formally - Highly organised cost effective model, highly effective at identifying cross-sell and up-sell opportunities.
• The reality - "You will be assimilated", but implementing a global enterprise BI system is easier to achieve in this model than any other.  Generally to have implemented this model means that the architecture and standards governance in the Enterprise must be top notch.

Before I go on to look at the impact that each of these operating models has on a BI strategy there is another important influencing factor.  Where in your organisation do your enterprise architects sit?  Obviously by this I don't mean the nice seats in the corner with the good windows and pot plants, but whats their reporting line?  Generally if they are within the IT organisation their effectiveness will be considerably reduced.  The tendency will be for the business to see them as "IT" so "what do they know about business" .  This was a very common theme at the EAC conference last week.   The risk is that while you may have "IT Architects" who do their best, the business is doing it's own Enterprise Architecture to their own tune.

How does this affect our BI strategy?  Well even if the EAs have a clear vision of the future if they don't have the authority to rigorously enforce this or they are seen as living in an ivory tower the reality is that over time new processes and capabilities will appear that bypass the central organisation.  Essentially your enterprise has reverted to the diversification model.  This becomes your problem when you try to build a BI capability for one model but the reality proves to be very different.  The advice here is form a good relationship with the EAs (you may even be in that group) but also look further afield to see the bigger picture.  Also remember that if they are doing their job well EAs will be having a tough time, they are often a voice of sanity in a enterprise trying to produce change for a better future, keeping that better future in mind is not an easy task when dealing with the day to day politics, especially as your actions may well render parts of the business redundant.  This was best summed up in a presentation last week by "When you are up to your arse in alligators, just remember you are there to drain the swamp".

So getting to the point, how should your BI strategy be influenced by the companies operating model? Lets start with the easy option, your enterprise is following a unification model, firstly, lucky you, someone else has already done most of the hard work of data integration and standardisation. In this model you can pretty much go to the shelf pick off a standard BI strategy of a single enterprise warehouse and stand a good chance of being successful.   All change should be planned and coordinated, where you just need to ensure you are plugged in at the right point.  Your technical implementation can also be considerably simplified by the reduced requirements for data integration and standardisation.  These are the models that the vendors love to tout as examples, being mostly successful and at the lower end of the cost range.

Now onto the slightly more problematic models, firstly replication.  The primary problem here is data integration and standardisation of entities not covered by the replicated processes. But the first question to ask here is do you need the "whole" view of the enterprise.  While the single view of the enterprise might be a nice to have and certainly required from the very top of the business, this may not even be needed at the lower levels.  Think of it as letting your BI strategy model follow that of the business.  Do you really need to go to the level of standardising elements such as customer address right across the enterprise?  As each business unit operates independently with it's own customer base what value are you adding by doing this?   So even if you build a single conceptual warehouse you can probably have dedicated model for each unit and only aggregate data at a level where it makes sense to do so.  While there is more effort involved in maintaining essentially separate solutions this may well be more successful that trying to force the business to change just to fit the niceties of your warehouse architecture.

Another slightly less problematic model is coordination.  Provided that everyone is sticking to the rules again most of the hard work for data standardisation and integration will have been done.  The problem here is how do we tailor reporting to suit local process variation?  The difficulty here is firstly there may be hidden data sources that support the localisation, these could even be the dreaded multi-thousand line spreadsheets, secondly you may well need to have a different view on the same data where local process practise adds a variation to the standard definition.  The latter problem can in part be resolved by moving away from the standard Kimball dimensional model to the more mature models using "Foundation layers" or similar techniques to abstract the dimensional model from the main data storage location.  Here each business unit can have it's own local dimensional model tailored to suit its local process variation.

Finally we reach diversification, this is really the wild west of BI and data warehousing, again you really need to ask the question "Do we need an Enterprise wide view?"  if the answer is yes the solution will be challenging not just from a standards and data perspective but technically as well with the associated costs of such complexity.  But the biggest hurdle will be getting business sponsorship for the project, your estimates will be put alongside example implementations using Qlikview in enterprises using a unification model with the inevitable questions about the extra cost.  Your first hurdle here is going to be getting the funding to just do the work to establish the scale of the problem and produce a governance and technical plan.  The only advice there is you have to do your best to present the vision of the possibilities and attempt to get a good executive sponsor who appreciates the scale of the problem.

While a standard approach here might be to start with the Kimball method of detailed requirements gathering followed by building a dimensional warehouse, this is going to be hugely costly and probably outpaced by the rate of change in the business.  So my best advice here is simple, start with a narrow scope, just a single process or area, but in all its diverse forms.  Use as much data abstraction as possible, expect change to your source systems.  Do just in time requirements gathering for your analytical layer, again requirements will change there is no point in documenting something in great detail only for it to be obsolete by the time you finish writing it.  A good approach to this problem is presented by Oracle in their paper "Information Management and Big Data,  Reference Architecture".  This is not an Oracle specific architecture and could be applied using a range of solutions. Deloitte present a very similar model in their paper on "How to build a successful BI strategy".  The most important role you are going to need in your team here is a top notch data modeller.  The abstract data model that constitutes the foundation layer or Enterprise Data Warehouse as Deloitte refer to it is a critical success factor.  The key here is to be able to focus on the similarities between data sources and processes, not the differences, while the detail and the differences are important it is all too easy to get bogged down in the detail and suffer from analytical paralysis.  Just make sure your data modeller is a "glass half full" personality.  The other key role will be your business analyst, they again need to focus at least initially on the "big picture" of what types of questions does the system need to answer, rather that diving into the detail of the actual questions or worse still report layouts and columns.

So in summary perhaps one of the questions you should ask before taking up a BI architecture role should be "What is your enterprise architecture model and governance process?", if they cannot answer or there is no enterprise model, be vary wary and look carefully at what you are taking on, you might be taking on the role of chief alligator keeper.