What next?

A colleague of mine PEG was recently blogging about the “value of information”. There are several ideas in his posts, one of them being that the value of the information to your organisation is largely influenced by the ‘freshness’ of the data you have access to. This aspect of data gives you a certain level of decision making power to determine 'what next' -the fresher the data, the more quickly we can react and potentially the more influence you can have on customer behavior.

Architecturally, we have established some fairly common approaches to dealing with data at a varying level of freshness. Traditional Business Intelligence (BI) solutions have been helping us look at historical data so that we can better understand long term business performance, customer behaviour, and therefore tune our business to perform better. Actions taken as a result of analysing BI data are usually optimisations to business processes, the introduction of new products or the removal of poorly performing products. These changes generally take weeks to months to implement as they require development changes to systems to recode, redeployment, and potentially training staff to take advantage of the new processes.

At the other end of the freshness scale, we have Business Activity Monitoring (BAM) which is all about real time event capture, aggregation, and monitoring via dashboards. BAM gives operational staff the ability to monitor performance metrics in real time and make decisions as business performance changes. Actions taken as a result of viewing BAM dashboards are usually immediate – e.g. to increase the number of call centre staff due to higher than average call rates, or to order additional stock due to higher than expected customer numbers.

There still seems to be some confusion with many customers on when to use BAM versus when to use BI, what is more valuable to the business? Whilst they both have their place in the enterprise, it’s the combination of data across the ‘freshness spectrum’ - BI, Operational, and BAM that can provide the complete picture needed to make well informed decisions.

A great example of how this concept has been applied is a story I heard about a casino. Every punter entering the casino would complete a short questionnaire, hand over some cash and be issued with a gaming card which they’d use at the various tables and pokies machines. Based on the completed questionnaire, the casino could establish a rough profile of the punter and estimate their expected spend. The gaming card provided an indication on what the punter was doing – betting on pokies, playing the tables, or cashing in for the night. If a punter looked like they were leaving prematurely (having not reached their estimated spend), casino staff would conveniently appear and offer a nights cheap accommodation or some credits to encourage the punter to stay on and keep gambling. The information involved in the decision to intercept the leaving punter is:

• BI data: Used to look at historical trends of punter profile against estimated spend. This data is likely to be used to influence questions asked to establish the profile and other major initiatives influenced by punter behaviour (e.g. what promotions were most effective in attracting people etc).

• Operational data: from the gaming system tracking bets, wins and losses etc, and operational data from the system handling hotel room reservations (where cheap accommodation is being offered).

• BAM data: real-time tracking of punter activity what each punter is currently doing in the gaming room and how much they’ve gambled.

So any single of type of data, whilst useful on its own, is not enough to take the decision to intercept. BI tells you what has happened historically, and therefore what to expect, BAM tells you what’s actually happening now, operational data tells you what you can offer – all three together tells us ‘what to do’.


To support this we need subsystems to manage this information and a means for integrating between them.

The BI data belongs in a data warehouse and data marts optimised for analytical slicing and dicing. There are plenty of BI solutions out there such as those offered by Oracle which provide end to end warehousing solutions complete with ETL components and analytical reporting front ends. Alternatively a basic star schema with a front end reporting tools such as Yellowfin or Crystal will do the trick also. The BI data will be updated frequently (e.g. nightly/hourly) via data feeds from operational data stores. This should be handled via change data capture (CDC) and a typical ETL technology such as Oracle Warehouse Builder or Talend which will provide the mechanisms to extract, transform and load data structures in an appropriate format for the warehouse.

The BAM system is responsible for capturing events fed from the gaming system as they occur in real time, and displaying on a dashboard for users to interpret - e.g. punter issued card, punter placed bet on roulette, punter cashing in etc. Events are usually published to the BAM system via asynchronous means such as JMS however this will depend entirely on the system creating the events. In some solutions the BAM event capture is ‘pull’ rather than ‘push’ – e.g. polling a database table for a change in source data. When particular events are captured, there may be a need for additional integration with other operational data stores to enrich the event, in this case to identify vacant rooms – e.g. a punter leaving event, a vacant room that is unlikely to be booked, and a reasonable level of confidence that the punter is likely to spend more. This integration will once again depend on the technology interface of the system being queried but many of the available BAM tools are well positioned to integrate with SOA environments, particularly where standards based web services are in use.

Lastly, the final data feed is the business user acting on the information in front of them which completes the cycle…

So this is an example where various types of information have been combined to answer the ‘what next’ question. All information is valuable, but the value can be increased when it's combined with other information elsewhere on the freshness scale.

Got any good examples where data fusion like this has been used effectively - let me know via comments.