The Nuts and bolts of BI vs. Data Science

By Samir Sharma

I’ve been to several events recently, some are data industry events i.e. data science or BI related, and some are vertical industry events for example insurance, retail etc.  The one thing that keeps coming up from people during the networking and canapes, is an interesting question. Something I wouldn’t have heard last year or the year before.  Basically, the question often put to me is:

What is the difference between Business Intelligence & Data Science?

It is a good question, as for several years most people have focused on the area of Business Intelligence and in the last year, data science has been getting a lot of press – mainly as it is synonymous with “Big Data”.  A term that still irks me, because, most people use it as a throwaway when they say oh yes you are in “big data!”

Over time, BI has become more popular with all companies, and has moved away from only being used by the blue-chip companies.  Mainly, because there are now many tools and software out there from a lot of different vendors, and choice as well as newer technology has driven the price of BI down.

This has made it a wider and accessible product to all from large to small companies.  Data Science on the other hand has started to become noteworthy, as folks are beginning to realise there is more than just Business Intelligence and they want to answer specific questions about their business, which may be future related – yes some would say a bit of wizardry!  We will get to that in a minute.

So, what are the differences between these two areas?

I think we probably need some definitions to kick things off, here goes:

Business Intelligence

Business intelligence (BI) is a technology-driven process for analysing data and presenting actionable information to help corporate executives, business managers and other end users make more informed business decisions. BI encompasses a wide variety of tools, applications and methodologies that enable organizations to collect data from internal systems and external sources, prepare it for analysis, develop and run queries against the data, and create reports, dashboards and data visualizations to make the analytical results available to corporate decision makers as well as operational workers. [techtarget]

Data Science

Data science, also known as data-driven science, is an interdisciplinary field about scientific methods, processes and systems to extract knowledge or insights from data in various forms, either structured or unstructured, which is a continuation of some of the data analysis fields such as statistics, classification, clustering, machine learning, data mining, and predictive analytics. [Wikipedia]

So now we have both definitions, I think there are some distinctions that one needs to make to go just a little further.

Business intelligence looks at what happened in the past and what is happening now.  So, for example, a sales team want to look at their performance, they will start looking at historical data and see how those trends are affecting sales today.

Data Science on the other hand is looking at what might happen tomorrow or next week (future related – back to the wizardry!).  For example, using data science a company can predict which products customers might buy in the future.  An elementary example, but, nonetheless one that can and has been possible using data science to drive more revenue and retention.

I did come across a very interesting matrix that gives a good some view of the differences, which I have amended to add my own thoughts.  It attempts to highlight where BI stops and Data Science takes off:

Just looking at the technologies, I would argue that there are also BI tools that can provide you with a “What-If” scenario, as it is looking back at historical data, and can therefore, look forward with a view.  So, there are elements of the above that I do disagree with.  Technology wise I do think they have different set of criteria, and the end goal can and often is different.

The other thing that strikes me as fundamental is the fact that as a data “scientist” one is working on a hypothesis (or a set of), and then presenting the facts that support these.  The cycle that supports this works in small iterations – one hypotheses is presented and then accepted or discarded, and then onto the next and so on.  Until the hypotheses have been validated and the facts are presented.

We also know that elements of data science have taken on the powerful application of machine learning.  This is the bit that is very different to BI.  Models that data scientists build are continuously learning and finding new insights.  The algorithms iteratively learn from the data, and if coupled with Artificial Intelligence, could well take on the brain of a computer and become more human like in its attributes.  I’ve strayed from the subject as one can do when talking about these things.

What makes data science so powerful is that the algorithms that are built, have an element of self-learning, and this is where it becomes fun!

Can the two areas be more like a symphony orchestra?

Too right they can!  The example provides a view of how both the BI team work with the data scientists interchangeably and collaboratively for the best outcome.

A logistics customer, looking at how they could deliver items within a predicted window, can mine the current data (using historical references), asking several questions about “what current compliance levels were in delivery based on a specific window in different locations, using different timings etc.”.  This showed that the possibility of what they wanted to achieve within a specific time window was there.

However, they then took that data and asked a specific question or posed a hypothesis, and this is where the data scientists came in with their algorithms and statistical modelling.  The algorithm learning over time with the data sets as they got bigger and more diverse, and as the output was analysed, the model could be tweaked to tighten the question and achieve the hypotheses.

Once this was done the output from the algorithm was then displayed in a rich visualisation tool translating the 1s & 0s into a story that the business could understand.

So in my opinion, to achieve the right notes they came together as an orchestra does with perfect timing!  Mind you at times, we did have the horn section make too much noise!

Where do we go from here?

You cannot have one without the other.  Can you imagine a company looking at data science when business users can’t even access their data fundamentals i.e. what is happening now and what has happened.  To move to a more data centric organisation BI & Data Science need to be brothers in arms.

Many companies rely on techniques driven by data science so that they aren’t always in reactive mode and start thinking with the future in mind, what business people would call predictive, prescriptive or proactive.  Therefore, and not to labour the point, BI is a weaker sibling than data science and can’t keep up in this game, unless technology factors change drastically.

To remain competitive and relevant in this brave new world, and to drive business strategies and objectives, companies should start naturally making the move (if they have adequate BI in place), to experimentation and data science.  It has and will be a game changing tactic, which will provide deeper insights into customers, drive innovation and shift growth into a new gear.  You may have a different view and that would be good to hear.

I am not surprised that more and more companies, as it is often reported in the press, are looking to invest in data science and big data strategies.  But, I do offer one thing that I have seen time and time again, and just not in the realms of data.

All the investment in the world into new technologies won’t help a company gain that competitive edge that they are looking for through data science, until they have changed internal behaviours and processes.  You might have bought the latest super-duper tech, which sits in the corner shining brightly, but if you don’t change or shift the culture it won’t be effective.  Many people seem to forget this, and falter at the alter when binding the marriage.

Until stakeholders are aware of the benefits, value and different ways of looking at things and “not doing the same thing every day, hoping for a different outcome”, then these technologies regardless of how many millions in growth they might add to the bottom line, will be stuck in the mire losing their shine!