thinking in geek tagline’s are so web2.0

Business Intelligence in Business Apps

I’ve been mulling over Ron Kohavi’s paper Integrating E-Commerce and Data Mining: Architecture and Challenge. In the space of applications that I work on – Business Intelligence and Analytics are becoming more and more important. It’s not just important that the system allows you to do whatever. (The classic post Blog entries for example). I need to be able to provide information about how users are interacting with the application.

• Who’s using what features?
• Where are the users coming from?
• How long does it take subset X of users to perform action Y?

Some of these are closely related to traditional web analytics, but many of them cannot be answered without application level event logging. Ron’s papers describe how Blue Martini did this within their e-commerce framework and what a huge advantage this gave them when doing BI. One of the insights of the paper above is that having a meta-model for your application is really important for later analysis. You need to know:

• When you changed the default answer for question X on the registration form?
• When did you add question Y to the checkout process?

If you keep a metamodel and use that for updating your live DB¹) then you can capture these changes as you change the data store. Ron’s team went further and built a data warehouse²) and ETL process automatically from the meta model – that was automatically updated as the live DB evolved.

I’ve been using some of these ideas independently for a long time (e.g. Event logging is a good idea) – but it’s the combination of of a meta model – including change history – and event logging that really provides the power here.

Short version. How I’m thinking about building business apps is changing. As I think about the business requirements, data model etc.. I’m now considering how we are going to capture/store/analyse the meta model and the business event stream. This seems like a significant evolution in my thinking.

1. (Blue Martini used a highly optimised/denormalised datastore for it’s e-commerce front end because of the performance requirements. [↩]
2. (Including slowly changing – aka type 2 – and hierarchical dimensions [↩]

You work with models ?!
Wicked.