|
|
One can reasonably argue that processes don’t produce results, people do. In and of itself a process does nothing. It takes people to engage in a process – for better or for worse – to produce something. On the other hand are quality pioneers like Edwards Deming who says: “Eighty-five percent of the reasons for failure to meet customer expectations are related to deficiencies in systems and process . . . rather than the employee.” “The role of management is to change the process rather than badger individuals to do better.” This quote does not take people completely out of the equation, but it places the focus squarely on the process rather than people.
Whether processes or people fail is not merely an academic question – it determines how we run our business. Every day we make dozens of business decisions. Both the decision maker and the information on which the decision is based are part of the decision making process. To make business decisions we to rely on information. Sometimes this information is based on “hard” data that has been collected, analyzed and interpreted – at other times we rely on “gut level instinct” that has been honed by years of experience. Regardless of where the information originates and how it was derived, the decision maker controls whether and how it used.
Decision makers are influenced by more than their perception of the information itself. Other factors, such as a vested interest in the outcome and one’s ability to understand the full significance of a piece of information, also play an important role. Bextra, Seroquel and Vioxx are just a few of the better known Pharma industry examples to illustrate how difficult the interpretation of data can be – and how much of its interpretation and perceived significance can be motivated by a vested interest. The drug dilution scandal involving Robert Courtney provides an excellent case study of what it takes before individual data points come together to tell a compelling story.
Neither people nor processes are perfect – simply because no one can really define what “perfection” means. No matter how well designed, processes are prone to failure when they do not keep pace with changes and when people lack adequate training, experience and time to do the work. Can a shrinking economy and vanishing jobs sustain processes that manage thousands of details? When people worry about their jobs, how do we decide which details to stop paying attention to? When people are overworked and pressed to do more than one job, can they still absorb all the information necessary to do everything well? When an emergency takes place, how many resources will it drain from other vital matters?
Let us leave the discussion of whether Six Sigma is a process, a methodology or a philosophy for another day and simply call it a “process” for making business decisions to improve the quality of our goods and services. This said, do the massive recalls from Toyota indicate that quality processes like Six Sigma are slow to adapt to a world in recession? Are they simply too resource intensive and complicated? Rather than blaming the process, is the company at fault for not having the right people and incentives in place to adapt processes to a changing world? What are the implications for those of us who collect, analyze and consume data to make business decisions?
Further Reading:
The Significance of Sigma: Toyota’s Lessons in Corporate Decision Making
http://pharma-bi.com/2010/02/the-significance-of-sigma-toyota%E2%80%99s-lessons-in-corporate-decision-making/
Visiting Toyota
PharmaManufacturing.com
http://www.pharmamanufacturing.com/articles/2009/032.html
Articles About ROBERT R. COURTNEY
http://topics.nytimes.com/topics/reference/timestopics/people/c/robert_r_courtney/index.html
Toyota’s Digital Disaster
In the Google era, how do you manage a product recall and a public-relations fiasco? Don’t do what Toyota’s done.
http://www.newsweek.com/id/232962
With the massive recall due to sudden acceleration problems, Toyota’s reputation for superior quality has suffered a black eye – if not more. The future will tell how serious this injury is and whether it represents the tip of an ominous iceberg. Sprinkled amongst the news coverage are hints that Toyota has known about accelerator problems for some time. From an outsider’s perspective this raises several questions about corporate decision making, including this one:
VOC or “Voice of the Customer” is a key concept in Six Sigma, the quality methodology used by Toyota and many other companies. Needless to say that with millions of customers, there are millions of opportunities for feedback – hence the potential for noise.
Wordplay aside, any communication from a customer contains some useful information, but not all feedback carries the same weight. For example, a broken radio most likely has less impact on car safety than a stuck gas pedal – but we can’t be sure until we have more information: the broken radio may be a symptom of an electrical problem that also affects the accelerator.
Therein lies the problem: how do we assign the “appropriate” value to the information we receive? How much effort and money do we put into researching the (hypothetical) “radio problem” versus other problems? How can we quickly assess whether the “radio problem” can turn into a “safety problem” that requires thorough attention? With the myriad of active and passive ways in which we can listen to customers, we need a good triaging system to help us separate critical information from information clutter.
While everyone can agree that data needs to be used “appropriately,” it is much more difficult to agree on what “appropriate use” actually means. Assuming for the moment that we can collect accurate data, what do we need to know in order to elevate an incident from “routine” to “requires immediate attention?” Here are several key factors that influence appropriate use:
- The ability to recognize the potential for significant harm
- The ability to draw a correlation between the incident and significant harm
- The ability to develop a solution to the problem
- The ability to implement a solution to the problem
- The ability to make that solution pay off in the long run
Each of these bullet points shares two characteristics: to accomplish them, we need good information as well as sound judgment – neither of which comes easily. This applies to all types of corporate decisions – whether we are dealing with product safety issues or the most profitable allocation of sales and marketing resources. The major differences between types of decisions typically revolve around their scale and the level of detail required to make a decision.
It is impractical to go through all the possible ways in which we can identify “appropriate” information. Instead, here are a few guidelines:
- Assess the potential harm
- Identify actionable information
- Prioritize timeliness, accuracy and budget
- Identify who needs to know what and when
- Incorporate the means to review requirements from time to time
Keeping these bullets in mind goes a long way toward selecting the tools and resources needed to supply appropriate information.
Additional Reading
Toyota knew of accelerator pedal problem in UK a year ago
From The Times
February 2, 2010
http://business.timesonline.co.uk/tol/business/industry_sectors/transport/article7011671.ece
Unintended Acceleration: Toyota Addresses the Issues
November 06, 2009 by Irv Miller
http://pressroom.toyota.com/pr/tms/our-point-of-view-post.aspx?id=2234
Wikipedia entry for Six Sigma, the quality control methodology used by Toyota and many other companies. Voice of the Customer (VOC) is a key concept of the Six Sigma methodology.
Recently, while working on input for a decision tree, I ran into a scenario that reminded me of the fact that we cannot improve a decision simply by applying a tool or technique. We also need good data.
Here is a hypothetical example: Let us assume we are a contractor who is evaluating a fixed bid contract. This contract will pay $115,000 if we accept a clause for liquidated damages of $50,000 in the event we do not meet some project conditions. We can remove this clause from the contract, but in that case it only pays $100,000.
From past experience we know that our project costs will fall somewhere between $80,000 and $90,000 and that the likelihood of coming in at the lower cost estimate is around 20%. This leaves an 80% chance that our costs will come in around $90,000. Looking at our current capabilities we estimate that we have a 90% chance of being able to meet all conditions and thus avoid having to pay damages.
Putting all of this into the decision tree pictured below, we conclude that accepting the liquidated damages clause is the better business decision.
 Decision Tree showing the EMV of two contract options
But how good is our estimate for avoiding damages? Can we really trust it? What data do we have to back it up? Have we really considered all the factors that can influence our estimate? After all, as the image below shows, if we are off by only 20 percentage points, the decision becomes a toss up.
 A decision tree showing what happens when we lower the assumption for avoiding damages from 90% to 70%
In a decision tree each chance node acts as a weighting factor, so it is worthwhile to pay special attention to events that are estimated to have a very high or very low chance of occurring. We want to be sure that we have good data to back up these optimistic (or pessimistic) numbers.
Of course it is not always feasible to gather all the data we need. Sometimes the data is too expensive given what is at stake, sometimes it is unavailable and sometimes the quality of the data is too unreliable for a given purpose. In that case, experience and judgment need to fill in the data holes. We also call this “making assumptions.”
When making assumptions, we should clearly identify them and decide what to do when one or more of them has to change. We need to
- identify which factors influence our assumptions
- determine how these factors influence the result
- be able to recognize when a significant change in our assumptions is needed
- have a process in place to handle these changes when they do occur.
No one can predict the future with certainty. But the more we understand the probabilities, the better prepared we are.
Ted Cuzzillo, the author behind the datadoodle blog, got me thinking about data details today. When do they matter and when do they distract from what matters?
Being a data analyst means that I love details: the more the better, so I can understand how they form the Big Picture. Intrinsically, I am drawn to graphs like this one:
 A scatter plot showing individual data points and 90th percentile reference lines with their respective values The spray of dots and their colors actually tell me something. They give me a feel for the data and point me toward what is driving the overall result. I can dig into individual data points and learn from them. On the other hand, many people need a more abstract view of the world – a view that boils down to the overall shape of things. After all, meaningful abstractions – like the graph below – are needed to make strategic, big picture decisions.
 A line graph averaging out the data points from the previous graph The graph above only plots 18 data points and connects them through a line to show the overall shape of the data. Of course, the more we abstract information, the more we loose the ability to derive meaningful insights.
In order to generate this line graph, I had to create bins into which I could group the many data points from the first graph. This means I now only have 18 data points from which to differentiate between the bottom 90% and the top 10% of the data. In the graph below, the numbers along each line indicate the number of records that have been binned to create each data point. As we can see from the 90th percentile reference lines below, the bottom 90% of the handful of data points in each section fall below 9 and 8 respectively.
 The same line graph as above, including 90th percentile reference lines However, the very first graph in this story shows us just how misleading the percentiles from the abstracted data are. According to the more detailed data, the 90th percentile values come out to 6.083 and 5.334 respectively. The abstracted values point in the right direction, but they are quite bit removed from the true values. The more detail we use, the closer we get to the truth.
When we use analysis tools like Tableau software, it becomes very important to keep our bearings about the data we are investigating. For example, we need to keep in mind that Tableau retrieves and calculates information based only on the data needed to generate the graph. That statement sounds really, duh, obvious. But we can get into trouble when we don’t think about it 
Let’s look at an example. Below are two graphs based on exactly the same underlying data – but why do the colors look different? Each graph appears to show a quadrant analysis that compares two web sites based on their search engine rank and trust.
 Graphs comparing two web sites based on their SEO rank and trust The difference lies in the way each graph is generated: the first graph really represents a set of eight data points, while the second graph represents two sets of four data points each – a subtle, but important distinction. The second graph shows the quadrants for each individual site using a separate scale for each site. This allows us to compare each site quadrant by quadrant without having to worry about one site having vastly more links than the other. In other words, we can answer questions like: which site did a better job of getting high quality links vs. low quality links?
The first graph combines the data for both sites and plots each quadrant on a scale for the combined data. If one site has many more links than the other site, it will skew the scale toward the higher linked site. In essence, we are comparing all eight quadrants against each other as opposed to comparing how each site performed on a particular quadrant.
The second graph therefore is the “correct” quadrant analysis if we want to compare each site quadrant by quadrant. But why even talk about the first graph?
That’s because in Tableau it may be tempting to generate the first graph to save time – especially when one is new to Tableau. We only have to drag the “Site Name” dimension onto the column shelf and, voila, we can show both sites next to each other. The problem is this: the shading is now determined based on all 8 data points together – rather than using a set of 4 data points for each individual site. This becomes obvious once we add color scales to the graphs:
 Graphs comparing two web sites based on their SEO rank & trust - includes color scales The first graph really does not compare the two sites to each other. Instead it takes a look at all the links for both sites combined and creates 8 data points from all those links. The second graph uses data from one site at a time. A small – but critical – difference.
While this example may seem trivial, it actually has deep implications when we deal with more complex visualizations. For example, when we use bins or when we filter records based on certain values, we may add misleading reference lines or create inaccurate charts – but that’s a topic for another day.
Have you ever had the “pleasure” of slogging through hundreds of feedback forms from a seminar or conference? Have you ever noticed how the mind seems to dwell on the negative comments, maybe even to the point that all the positives seem to loose their luster? That’s when crunching actual survey numbers can help put things into perspective: either there really were problems or we are about to fall prey to the naysayers and constant critics.
For example, this heat map makes it obvious that Session C was the least popular event in this conference – but notice also that the range of scores is fairly close together. In this example, attendees used a 5 point scale, with 1 being the least favorable score and 5 being the most favorable score. Seeing that the lowest average score was above 4 tells us that, overall, attendees were quite happy with this conference.
 A heatmap showing how each individual session was rated along multiple criteria -- click on the picture to enlarge it Of course we also want to understand what worked well and what didn’t work so well. Let us take a more detailed look at Session C.
 Click the picture to enlarge it Very quickly we can tell that the audience was critical of the session content and how it was presented rather than the speaker’s knowledge about the subject. More than 90% felt that the speaker had an adequate background to present on this topic. But less than 75% were happy with the way the information was presented and how it related to their job. A few people awarded low scores of 2 and 1, but it is encouraging that these low scores came from fewer than 10% of the audience members.
Just to provide a contrast, let’s also take a look at Session B, clearly the favorite of this event.
 Individual Scores for Session B There is no doubt that the audience rated this Session very highly across the board: 74% or more audience members awarded it the highest score across all measures. Notice that this session, too, received a handful of low scores.
As we can see here, sometimes we only need a few pictures to gain useful insights. No doubt, if this were a more involved customer survey, we would need more and very likely different graphs to discern the finer points of attitudes and perceptions — especially when such information is tracked over time. But pictures like these are a good start to find out where to focus such additional efforts.
Last week I wrote about the need to expand our visual vocabulary in line with software that allows us to graph complex data relationships and events in ever more meaningful ways. Let me expand on this point with an example.
Just about any introductory statistics course covers the draft lottery of the Vietnam War era as a case study for determining whether events occurred by chance or whether they were subject to some significant influence. Typically this case study involves a scatter plot and discussions about statistical significance, p-values and regression lines.
The section of the draft lottery we are discussing here involved the following process: Each Birthday was given a number from 1 to 366 (including leap day) so that 1 = Jan 1st, 2 = Jan 2nd, 3 = Jan 3rd and so on until 366 = Dec 31st. Each Birthday number was written on a piece of paper, put in a plastic capsule and then in a shoebox from which each was later drawn one at a time. The Draft Number represents the order in which each birthday was drawn from the shoebox.
The scatter plot below shows Birthdays vs. Draft Numbers along with a trend line to indicate the relationship between the two. The casual observer may not notice the relative sparsity of data points in the upper right hand quadrant, let alone grasp the significance of it. It takes careful observation to notice that a significant number of birthdays that occur later in the year have lower draft numbers and visa versa.
 Scatter plot and trend line showing Birthday vs. Draft Number. P < 0.0001 - Click the picture to enlarge it. During business discussions or presentations we often do not have time to explain statistical models and their implications. Sometimes we need to find more intuitive means to get our point across quickly and effectively. When constructed properly, visual displays can convey a lot of information very quickly.
To illustrate the results from the Draft Lottery more intuitively, I divided all birthdays into three equal groups: Group 1 includes days 1 – 122, Group 2 includes days 123 to 244 and Group 3 includes days 245 to 366. I then plotted the corresponding draft numbers in each Birthday Group.
At a glance one can see that the Birthdays tend to cluster toward the upper end of the Draft Numbers range in Group 1 and toward the lower end of the Draft Numbers range in Group 3. There is no need to talk about p-values and regression lines, just a quick note pointing out that Birthdays in the early part of the year had a better chance of receiving a high draft number than Birthdays that occurred later in the year.
 Draft Numbers by Birthday Group. Click the picture to enlarge it. An alternative representation may include reference lines for quartiles as in this illustration:
 Draft Numbers by Birthday Group - including reference lines for quartiles. Click the picture to enlarge it. Business decisions, of course, are based on more than fancy pictures and we need to be able to back things up through detailed analysis. But when time is short and we need to make our point quickly, the second graph helps us out much more than the scatter plot.
Acknowledgements:
A big “Thank You” goes to the authors of “Online Statistics: An Interactive Multimedia Course of Study” for providing the data and for the inspiration to visualize it in a format other than a scatter plot. This free online course was developed at Rice University, University of Houston, Clear Lake, and University of Houston, Downtown, with partial support from the National Science Foundation.
All graphs were generated using Tableau software.
During several conversations recently the following comment came up: “What’s the big deal with visual analytics? It’s just a bunch of pretty pictures!” It took a while, but it finally dawned on me that we have reached the Xerox-GUI-Macintosh stage for data analysis. The early versions of a graphical user interface (GUI) which were developed at what was then called Xerox-PARC were no commercial success until Macintosh – now better known as Apple – created a computer that allowed everyone to point and click rather than write arcane computer instructions.
While point-and-click is much easier than writing code, it still requires computer users to become familiar with what the computer can do and how to accomplish various tasks. It also requires standards about where to click and what should happen when certain actions are taken. All this knowledge and these standards had to develop over time and often through trial and error.
In some ways we have reached a similar stage for data analysis: visual analytics provides a new language through which non-analysts can explore and answer business questions. It frees the non-technical user from the analytic equivalent of writing code, that is, it frees them from the need to learn how to create graphics that – until now – required significant technical knowledge to generate.
As with any new technology, some mayhem ensues: we have to gain experience and learn through practice. We need to become fluent in the appropriate use of less familiar – yet oddly intuitive – graphing techniques like sparklines, heatmaps or small multiples. In short, we must develop a visual vocabulary beyond the bar charts and line graphs we know from Excel or PowerPoint.
And this brings me back to the Xerox-GUI-Macintosh comment from the beginning: as in those early days of learning the language of icons and point-and-click, we have now reached the point where more powerful ways of encoding data in a visual format is available to the lay person. Just as with the graphical user interface, we will some day look back and say “I can’t imagine a world without seeing data in pictures.” For at least a little while, those of us in the analyst professions need to act as interpreters and guides to those who are learning this new language. Sooner rather than later we will all get there.
For several years I have participated in the Heartland Association of Research Professionals (HARP), a non-profit organization that provides educational and networking opportunities for nearly 200 professionals engaged in clinical research and related fields.
Recently we surveyed current and prospective HARP members to gather feedback regarding the services we provide. This gave me a nice opportunity to learn more about Tableau. While I did encounter some formatting issues, Tableau provided a quick and easy way to explore survey results.
Quite useful was the ability to set up all the graphs, tables and dashboards while the survey data was being collected. Periodically I could update everything simply by refreshing the underlying data. This allowed me to easily keep my fellow board members in the loop while data was rolling in. It also allowed me to send reminders to survey participants using actual completion times to assure them that participation would really take only take a few minutes of their time.
Questions that allow for multiple answers can be a bit tricky to analyze, but visualizing the results with Tableau made that analysis much easier. The example here illustrates the various research certifications held by HARP members. This question focused on research certification as opposed to certification in other domains.
Below is an example using graphical clues and annotation to illustrate which of four networking related services were used by current and prospective HARP members. I cannot imagine anyone who would want to read a written description of these results. A picture communicates this much better and without the proverbial “thousand words.”
To improve readability, Tableau provides a great variety of formatting and labeling options such as turning titles, column and row headings on or off.
Example 1: Title turned on, Row Headings turned off:
Example 2: Title turned off, Row Headings turned on:
From a communications standpoint, it would have been nice if there had been a way to include all options of the five point scale in this graph. The fact that no one selected “Somewhat Unimportant” nor “Unimportant” would be much easier to see if I did not have to talk about it in a comment.
Using nothing more than zip codes as geographic coding data, Tableau provided us with a quick idea about the geographic dispersion of our membership – very handy, since picking a good meeting location is always a tricky topic.
The two previous articles of this series explored how to prepare for the future by prioritizing and streamlining our work. This article looks at what it takes to get there in terms of goals, skills and people. At this stage we should already have a very good idea about
- What information the organization needs
- Who needs to see what information
- How to prioritize information deliverables
- How often we need to update information
The third step to a successful BI future requires that we
- Set Goals
- Find out what skills we need
- Determine who needs to do what
Of course we also need a budget and organizational support, but for today, we will focus just on Goals, Skills and People.
Setting Appropriate Goals
Much of the discussion around goal setting deals with managing and motivating people to strive for a common goal. It includes such well known concepts as
Peter Drucker’s MBO (“Management by Objectives”) or
SWOT Analysis based on the work by Albert Humphrey or
“The 7 Habits of Highly Effective People” by Steven Covey or
BHAG (“Big Hairy Audacious Goal”) by James Collins and Jerry Porras.
For our purposes today let us focus on the acronym S.M.A.R.T. It’s easy to remember and useful for developing the kinds of delivery goals that a BI Manager needs to set and implement. Depending on the author, the letters S.M.A.R.T. stand for slightly different words. One such set of words is:
S: Specific
M: Measurable
A: Achievable
R: Relevant
T: Time bound.
A very nice one page summary about setting S.M.A.R.T. goals can be found here.
A hypothetical goal could be: “In six months we will need to deliver sales goals and compensation measures for 50 sales reps, 5 managers and a VP of Sales.” Let’s see how S.M.A.R.T. it is:
Specific: deliver sales goals and compensation measures for 50 reps, 5 managers and a VP of Sales
Measurable: at the end we either can deliver something or we cannot
Achievable: we know that sales goals and compensation measures are required – failing to develop them simply is not an option. We also know that achieving goals usually depends on finding the right balance between cost, quality and time. Finding that balance is the key to making this goal achievable.
Relevant: a company does not survive long without sales goals and compensation measures
Time Bound: in six months
It appears that we really do have a S.M.A.R.T. goal on our hands.
Identifying the Right Skills
Today’s business analysis software makes it much easier to perform data analysis and performance tracking without requiring programming skills. But software alone does not provide everything we need. Appropriate data has to be collected, processed and made available in a timely manner. To stick with our hypothetical example of providing sales goals, here is an example list of several skills we will need:
- Understanding of the market place and business goals
- Knowledge about what types of data are needed
- Knowledge about how the data is collected and processed
- Understanding the shortcomings of the data and how to address them
- Ability to make the necessary data available and accessible
- Ability to distribute information in appropriate formats
- Analytical skills to develop meaningful measures, metrics and conclusions
In our hypothetical example we may already have someone on our team who knows all the necessary data sources. Maybe this person already knows which information from hospitals, doctor offices, contracting and claims systems, SFA systems and a myriad of other sources has to be combined to develop meaningful goals. Maybe this person already knows how to develop compensation formulas that appropriately reflect business goals using data that is actually available. Maybe this person also has the skills to process and analyze data and to make it available to everyone who needs it.
In reality it is much more likely that a team of internal and external resources needs to work together to provide the necessary information and insights.
Finding the Right People
The actual list of skills depends on the individual situation, of course. Sometimes it is useful to develop and maintain the necessary skill sets within an organization, at other times it makes sense to hire specialists. Budgets, time requirements, available skill sets and long term vision for the organization will help determine the appropriate mix of in-house talent and external resources.
As a whole, the team not only needs technical skills (as outlined above), but also management and communication skills. To effectively work together, team members need to
- Be able to identify critical issues and communicate them appropriately
- Understand each other’s requirements
- Know their respective roles in the project
- Possess project and time management skills
- Possess people and team building skills
Ideally, our team combines representation from the business side as well as the information side. For example, developing sales goals requires
- input from all levels of the sales organization
- input from product management and other strategic stake holders
- input from HR and other relevant support functions
- buy-in from upper management
- the ability of the information team (BI, IT, vendors, etc.) to deliver
- appropriate, accurate and timely data
- systems to make data available
- analysis and actionable insights
Managing Growth: Reprise
We began this series of blog posts with the question of how to manage BI growth. We started with ways to prioritize deliverables, continued with ways to streamline what the BI team needs to deliver and ended with an exploration of the skills and people necessary to deliver what is needed (the article you are reading now).
Managing BI growth requires constant attention since this process never really ends. For each deliverable we are asked to provide, it is therefore necessary to develop criteria that allow us to determine when it needs to either stop or evolve. We need to balance available head count and the need for professional development and advancement of individual team members with the information delivery needs of the organization. We need to keep track of technical developments and the services provided by various data vendors.
As BI managers we periodically need to step away from the daily fires in order to plan ahead. The steps outlined in this series provide a starting point for doing so.
|
|
