Visual Analytics: Breaching the New Frontier in Business Analysis

Ever since attending a data mining workshop in the late 1990’s I wondered how soon software would evolve to the point where someone could draw meaningful conclusions from data without needing a Ph.D. in statistics or extensive database and query tool experience.   It now appears that we have crossed the frontier into this new era of informed decision making.

Software companies like Tibco/Spotfire and Tableau are putting visual analytics front and center when it comes to helping businesses find out where their bottom line hurts and where opportunities may be hiding.  A recent article by Ted Cuzzillo for The Data Warehousing InstituteTM talks about a “new breed of BI Analyst” who wants to dive into the data, try things out, see what works and explore it in an interactive, almost intuitive, manner.  Enterprise blogger Michael Vizard, who writes for the Ziff Davis suite of publications, even foresees “The Demise of Report Writers and Spreadsheet Jockeys.”

Too good to be true?

Remember the promise of the “E.I.S.”, the Executive Information System, which was supposed to put all the answers at the fingertips of executives and management?  Once the novelty wore off, executives and managers asked their administrative staff to pull data and print out reports from their company’s E.I.S.  When questions arose, the IT or Business Information departments still needed to provide answers.

I don’t mean to rain on anyone’s parade: visual analytics is the future and I enjoy using tools such as Tableau (see my review here).  Visual exploration opens the door to much faster insight and understanding of what’s happening in a business.  Rather, this is meant as a cautionary note not to get caught up in the belief that a tool alone can solve our information problems and turn someone into an expert analyst with all the correct answers.  The risk with visual analytics lies in the fact that it appears to turn data analysis into child’s play, when, in fact, real work has to be done.

Tackling the hidden hazards of visual analytics

Let’s face it: to draw useful conclusions one has to deal with the fact that data is imperfect, that factors beyond our control affect the quality of data and that sometimes good judgment is needed to discern the meaningful from inconsequential.

An analyst has to understand the data behind each graph and know how it is collected, processed and presented.  When is data updated? Why could data be missing or incorrect? When is an unusual observation due to problems with the underlying data rather than actual performance of the business?  These are just some of the factors that may influence the outcome of an analysis, visual or not.

Granted, if the underlying data tables are well designed and if good data quality processes are in place, visual analysis becomes much safer for someone who doesn’t have time to worry about data schemas, update cycles and data scrubbing.  None the less, the person performing the analysis still has to know how to question the underlying data or risk reaching misinformed conclusions.

Visualization tools often make it easy to group data and to create new summaries on the fly.  Discerning what really drives results can be tricky when dealing with such summaries and groupings.  For instance, when drugs are used to treat multiple indications, how do we know that the data group we are exploring really contains all the relevant elements?  There’s something to be said for good meta data management and clear definitions for the metrics used by decision makers.

Graphs and charts are simply clutter unless one has the good judgment of knowing which blips are worth exploring and which are simply noise.  Some outliers point to real issues while others are merely distortions.  Knowing which is which requires business knowledge and a feel for how data behaves.  For instance, a spike in unit sales has very different implications for the bottom line when high unit sales are achieved through heavy discounting for major customers as opposed to gaining new customers at full price.

Visual analytics is no longer the next frontier

The emergence of visual analytics feels a little bit like going from command line DOS to the visual interface of Windows.  The further away we get from knowing about the inner workings of a computer – or in this case “the data” – the easier it becomes for errors to slip past our awareness because we don’t know what questions to ask or what can get us into trouble. Just ask any computer security expert how often people neglect to install or maintain intrusion protections on their computers.  Yet, nowadays no one can imagine life without computers anymore.

Today’s abilities to easily pull information together from various sources, to create graphs on the fly and to select individual data points for further analysis would be impossible without the ability to remove ourselves from the tedious details of collecting, transforming, staging and retrieving data.  Without the ability to gain distance from the inner workings of data we would drown in information overload – which really is no better than having to run a business by gut feeling and dead reckoning.

The real power of visual analytics comes to the fore because an analyst now has the tools to combine business knowledge and data expertise without having to spend hours preparing code or staging data in more effective ways.  That time can now be spent doing analysis and deriving value from the data.

However, with power comes responsibility.  Just because today’s tools remove us from the tedium of data preparation doesn’t mean we abdicate our responsibility for analyzing what’s under the hood.  Instead, we have to develop QC methods and standards that allow us to recognize when something goes awry.

Personally, I am happy to see that business intelligence applications and visualization are merging together.  Point and click frees my inner data geek from having to write tons of code and allows me to get right to the fun part: seeing what I’m dealing with, understanding how the pieces fit together and figuring out answers. Now I’m just waiting for the day when we can say “Computer, tell me how to answer this question!”  That’s the day I get to retire 🙂

2 comments to Visual Analytics: Breaching the New Frontier in Business Analysis

  • Dan:

    thank you for your comment. You are right to say that the speed with which we can point out the question and with which we can explore answers are key success factors. While CEOs or senior managers may not want to spend much time digging into the details, visual analytics at least gives them the ability to fine tune their “gut feeling” about a situation much sooner than if they had to wait for someone else to provide more insight.

    Stephen Few’s book is on my “summer reading list” – for reader’s unfamiliar with his work, here is a link to his web site/blog

  • I think the real benefit is the “rapid” interaction with the data set. I travel a lot and carry a Moleskine notebook with the picture of a time series chart with not words on it. Whenever I’m in a bar or restaurant sitting next to someone I show them the picture of the chart and ask them, “What’s the first question that comes to mind?”

    The context of time changes from seconds to decades. A heart surgeon’s context was seconds….a government statistician’s context was decades….but the question is nearly always the same…Why?

    Rapid visual analytics helps you get from the QUESTION to INSIGHT very quickly. See Stephen Few’s new book “Now You See It” for a very good review of the whole topic.