Pharma BI

Decision Making During Turmoil: How Well Prepared Are We?

Christine Muser — Thu, 11 Mar 2010 00:34:56 +0000

In order to make profitable decisions, we need good information. Whether we base our decisions on sales, customer perceptions or the number of widgets we shipped last month, our information comes from some system that collects and measures relevant data for us.

In my Six Sigma Black Belt class we recently discussed the challenges of developing a meaningful measurement system. As usual, the theory sounds easy – until it hits the road of reality. A very simple class room exercise illustrated that point neatly: our instructor had gone through the effort of individually placing twenty M&M candies into twenty numbered plastic bags and then asked us to “accept” or “reject” each M&M based on three criteria. The criteria were written down and no additional verbal cues were given nor did we have a “master” M&M on which to base our judgment.

We realized very quickly that these criteria were not nearly as clear cut as they appeared to be. For example, one criterion specified that the letter “m” on the candy should be “100% visible.” Sounds clear cut, right? After all, is has a numeric qualifier to help us make our decision! Reality check: have you ever looked at an M&M up close? The next time you do, look for tiny spots where the white ink is thin enough for the underlying color of the candy to bleed through the letter “m.” Question: if the entire outline of the letter “m” appears on the candy but these little flecks of color are bleeding through, does this mean that the “m” is no longer 100% visible?

The graph below shows the result of the M&M exercise. It illustrates just how far apart the judgment of perfectly reasonable people can be when they are asked to interpret someone else’s instructions. The left hand graph shows how much each team agreed with itself after reviewing all 20 candies twice in a row. The right hand graph shows how much each team agreed with an external standard for evaluating the candies. The fact that the two red lines barely line up with each other illustrates just how far apart the two teams were with their assessment of the same group of M&Ms.

M&M Attribute Agreement Analysis - click the picture to enlarge it.

The real issue, of course, has nothing to do with the candy and how it looks. The bigger point lies in something the Six Sigma folks call “operational definitions” and how we use them. The M&M example illustrates just how unpredictable individual judgments can be and how much training and feedback may be required before team members reach similar conclusions – which, in turn, will allow the team to work toward a common goal.

As the M&M example shows, developing operational definitions can be tricky. Definitions may be less clear cut than we think. We have a limited amount of time in which to develop them. In group settings, we also have to figure in personalities and hidden agendas. Good leadership and negotiation skills are needed to keep everyone focused without suppressing critical input. In the world of sales and marketing we have the additional challenge of dealing with missing and incomplete data. While statistical models go a long way toward filling in the picture, they are difficult to explain and are not always accepted by those whose paycheck depends on them or by those whose experience seems to indicate something else.

Some ideas for dealing with all this will be the subject of future posts. For today I simply want to ask these questions: with so many changes in the health care marketplace, how well are we prepared to make decisions? Which operational definitions do we need to add, update or toss out in order to ensure good decisions for the future?

P.S.: Additional Information About The M&M Graph

This data mimics the results from a Measuring System Audit (MSA) project with M&M candies. The assignment was to inspect 20 pieces of candy and to determine whether each met these three criteria:

1: the letter ‘m’ is 100% visible
2: the ink for the letter ‘m’ is not smudged
3: there are no chips

Only these written criteria were given. Neither team received additional instructions nor a “Master” against which to evaluate the candy. Each team was asked to review the candies in two rounds. During the first round, Team 2 decided to fail all 20 pieces of candy, hence that team’s low rate of agreement.

Conclusion: gaining agreement about operational definitions is critical. Make sure that everyone has the same training and verify that everyone in a decision making role can reach decisions that support the established goal. Repeat training and offer opportunities for feedback & refinement of criteria.

Toyota: Did Six Sigma Fail or Did People Fail?

Christine Muser — Thu, 04 Feb 2010 21:07:03 +0000

One can reasonably argue that processes don’t produce results, people do. In and of itself a process does nothing. It takes people to engage in a process – for better or for worse – to produce something. On the other hand are quality pioneers like Edwards Deming who says: “Eighty-five percent of the reasons for failure to meet customer expectations are related to deficiencies in systems and process . . . rather than the employee.” “The role of management is to change the process rather than badger individuals to do better.” This quote does not take people completely out of the equation, but it places the focus squarely on the process rather than people.

Whether processes or people fail is not merely an academic question – it determines how we run our business. Every day we make dozens of business decisions. Both the decision maker and the information on which the decision is based are part of the decision making process. To make business decisions we to rely on information. Sometimes this information is based on “hard” data that has been collected, analyzed and interpreted – at other times we rely on “gut level instinct” that has been honed by years of experience. Regardless of where the information originates and how it was derived, the decision maker controls whether and how it used.

Decision makers are influenced by more than their perception of the information itself. Other factors, such as a vested interest in the outcome and one’s ability to understand the full significance of a piece of information, also play an important role. Bextra, Seroquel and Vioxx are just a few of the better known Pharma industry examples to illustrate how difficult the interpretation of data can be – and how much of its interpretation and perceived significance can be motivated by a vested interest. The drug dilution scandal involving Robert Courtney provides an excellent case study of what it takes before individual data points come together to tell a compelling story.

Neither people nor processes are perfect – simply because no one can really define what “perfection” means. No matter how well designed, processes are prone to failure when they do not keep pace with changes and when people lack adequate training, experience and time to do the work. Can a shrinking economy and vanishing jobs sustain processes that manage thousands of details? When people worry about their jobs, how do we decide which details to stop paying attention to? When people are overworked and pressed to do more than one job, can they still absorb all the information necessary to do everything well? When an emergency takes place, how many resources will it drain from other vital matters?

Let us leave the discussion of whether Six Sigma is a process, a methodology or a philosophy for another day and simply call it a “process” for making business decisions to improve the quality of our goods and services. This said, do the massive recalls from Toyota indicate that quality processes like Six Sigma are slow to adapt to a world in recession? Are they simply too resource intensive and complicated? Rather than blaming the process, is the company at fault for not having the right people and incentives in place to adapt processes to a changing world? What are the implications for those of us who collect, analyze and consume data to make business decisions?

The Significance of Sigma: Toyota’s Lessons in Corporate Decision Making

Christine Muser — Tue, 02 Feb 2010 23:39:05 +0000

With the massive recall due to sudden acceleration problems, Toyota’s reputation for superior quality has suffered a black eye – if not more. The future will tell how serious this injury is and whether it represents the tip of an ominous iceberg. Sprinkled amongst the news coverage are hints that Toyota has known about accelerator problems for some time. From an outsider’s perspective this raises several questions about corporate decision making, including this one:

How does one differentiate between the “voice of the customer” and the “noise of the customer?”

VOC or “Voice of the Customer” is a key concept in Six Sigma, the quality methodology used by Toyota and many other companies. Needless to say that with millions of customers, there are millions of opportunities for feedback – hence the potential for noise.

Wordplay aside, any communication from a customer contains some useful information, but not all feedback carries the same weight. For example, a broken radio most likely has less impact on car safety than a stuck gas pedal – but we can’t be sure until we have more information: the broken radio may be a symptom of an electrical problem that also affects the accelerator.

Therein lies the problem: how do we assign the “appropriate” value to the information we receive? How much effort and money do we put into researching the (hypothetical) “radio problem” versus other problems? How can we quickly assess whether the “radio problem” can turn into a “safety problem” that requires thorough attention? With the myriad of active and passive ways in which we can listen to customers, we need a good triaging system to help us separate critical information from information clutter.

While everyone can agree that data needs to be used “appropriately,” it is much more difficult to agree on what “appropriate use” actually means. Assuming for the moment that we can collect accurate data, what do we need to know in order to elevate an incident from “routine” to “requires immediate attention?” Here are several key factors that influence appropriate use:

The ability to recognize the potential for significant harm
The ability to draw a correlation between the incident and significant harm
The ability to develop a solution to the problem
The ability to implement a solution to the problem
The ability to make that solution pay off in the long run

Each of these bullet points shares two characteristics: to accomplish them, we need good information as well as sound judgment – neither of which comes easily. This applies to all types of corporate decisions – whether we are dealing with product safety issues or the most profitable allocation of sales and marketing resources. The major differences between types of decisions typically revolve around their scale and the level of detail required to make a decision.

It is impractical to go through all the possible ways in which we can identify “appropriate” information. Instead, here are a few guidelines:

Assess the potential harm
Identify actionable information
Prioritize timeliness, accuracy and budget
Identify who needs to know what and when
Incorporate the means to review requirements from time to time

Keeping these bullets in mind goes a long way toward selecting the tools and resources needed to supply appropriate information.

Additional Reading

Toyota knew of accelerator pedal problem in UK a year ago
From The Times
February 2, 2010

http://business.timesonline.co.uk/tol/business/industry_sectors/transport/article7011671.ece

Unintended Acceleration: Toyota Addresses the Issues
November 06, 2009 by Irv Miller

http://pressroom.toyota.com/pr/tms/our-point-of-view-post.aspx?id=2234

Wikipedia entry for Six Sigma, the quality control methodology used by Toyota and many other companies. Voice of the Customer (VOC) is a key concept of the Six Sigma methodology.

Why We Need Good Data

Christine Muser — Thu, 10 Dec 2009 17:54:59 +0000

Recently, while working on input for a decision tree, I ran into a scenario that reminded me of the fact that we cannot improve a decision simply by applying a tool or technique. We also need good data.

Here is a hypothetical example: Let us assume we are a contractor who is evaluating a fixed bid contract. This contract will pay $115,000 if we accept a clause for liquidated damages of $50,000 in the event we do not meet some project conditions. We can remove this clause from the contract, but in that case it only pays $100,000.

From past experience we know that our project costs will fall somewhere between $80,000 and $90,000 and that the likelihood of coming in at the lower cost estimate is around 20%. This leaves an 80% chance that our costs will come in around $90,000. Looking at our current capabilities we estimate that we have a 90% chance of being able to meet all conditions and thus avoid having to pay damages.

Putting all of this into the decision tree pictured below, we conclude that accepting the liquidated damages clause is the better business decision.

Decision Tree showing the EMV of two contract options

But how good is our estimate for avoiding damages? Can we really trust it? What data do we have to back it up? Have we really considered all the factors that can influence our estimate? After all, as the image below shows, if we are off by only 20 percentage points, the decision becomes a toss up.

A decision tree showing what happens when we lower the assumption for avoiding damages from 90% to 70%

In a decision tree each chance node acts as a weighting factor, so it is worthwhile to pay special attention to events that are estimated to have a very high or very low chance of occurring. We want to be sure that we have good data to back up these optimistic (or pessimistic) numbers.

Of course it is not always feasible to gather all the data we need. Sometimes the data is too expensive given what is at stake, sometimes it is unavailable and sometimes the quality of the data is too unreliable for a given purpose. In that case, experience and judgment need to fill in the data holes. We also call this “making assumptions.”

When making assumptions, we should clearly identify them and decide what to do when one or more of them has to change. We need to

identify which factors influence our assumptions
determine how these factors influence the result
be able to recognize when a significant change in our assumptions is needed
have a process in place to handle these changes when they do occur.

No one can predict the future with certainty. But the more we understand the probabilities, the better prepared we are.

When Data Details Matter

Christine Muser — Tue, 24 Nov 2009 00:37:10 +0000

Ted Cuzzillo, the author behind the datadoodle blog, got me thinking about data details today. When do they matter and when do they distract from what matters?

Being a data analyst means that I love details: the more the better, so I can understand how they form the Big Picture. Intrinsically, I am drawn to graphs like this one:

A scatter plot showing individual data points and 90th percentile reference lines with their respective values

The spray of dots and their colors actually tell me something. They give me a feel for the data and point me toward what is driving the overall result. I can dig into individual data points and learn from them. On the other hand, many people need a more abstract view of the world – a view that boils down to the overall shape of things. After all, meaningful abstractions – like the graph below – are needed to make strategic, big picture decisions.

A line graph averaging out the data points from the previous graph

The graph above only plots 18 data points and connects them through a line to show the overall shape of the data. Of course, the more we abstract information, the more we loose the ability to derive meaningful insights.

In order to generate this line graph, I had to create bins into which I could group the many data points from the first graph. This means I now only have 18 data points from which to differentiate between the bottom 90% and the top 10% of the data. In the graph below, the numbers along each line indicate the number of records that have been binned to create each data point. As we can see from the 90^th percentile reference lines below, the bottom 90% of the handful of data points in each section fall below 9 and 8 respectively.

The same line graph as above, including 90th percentile reference lines

However, the very first graph in this story shows us just how misleading the percentiles from the abstracted data are. According to the more detailed data, the 90^th percentile values come out to 6.083 and 5.334 respectively. The abstracted values point in the right direction, but they are quite bit removed from the true values. The more detail we use, the closer we get to the truth.

How to Create a Misleading Quadrant Analysis – by Accident

Christine Muser — Mon, 23 Nov 2009 22:23:29 +0000

When we use analysis tools like Tableau software, it becomes very important to keep our bearings about the data we are investigating. For example, we need to keep in mind that Tableau retrieves and calculates information based only on the data needed to generate the graph. That statement sounds really, duh, obvious. But we can get into trouble when we don’t think about it

Let’s look at an example. Below are two graphs based on exactly the same underlying data – but why do the colors look different? Each graph appears to show a quadrant analysis that compares two web sites based on their search engine rank and trust.

Graphs comparing two web sites based on their SEO rank and trust

The difference lies in the way each graph is generated: the first graph really represents a set of eight data points, while the second graph represents two sets of four data points each – a subtle, but important distinction. The second graph shows the quadrants for each individual site using a separate scale for each site. This allows us to compare each site quadrant by quadrant without having to worry about one site having vastly more links than the other. In other words, we can answer questions like: which site did a better job of getting high quality links vs. low quality links?

The first graph combines the data for both sites and plots each quadrant on a scale for the combined data. If one site has many more links than the other site, it will skew the scale toward the higher linked site. In essence, we are comparing all eight quadrants against each other as opposed to comparing how each site performed on a particular quadrant.

The second graph therefore is the “correct” quadrant analysis if we want to compare each site quadrant by quadrant. But why even talk about the first graph?

That’s because in Tableau it may be tempting to generate the first graph to save time – especially when one is new to Tableau. We only have to drag the “Site Name” dimension onto the column shelf and, voila, we can show both sites next to each other. The problem is this: the shading is now determined based on all 8 data points together – rather than using a set of 4 data points for each individual site. This becomes obvious once we add color scales to the graphs:

Graphs comparing two web sites based on their SEO rank & trust - includes color scales

The first graph really does not compare the two sites to each other. Instead it takes a look at all the links for both sites combined and creates 8 data points from all those links. The second graph uses data from one site at a time. A small – but critical – difference.

While this example may seem trivial, it actually has deep implications when we deal with more complex visualizations. For example, when we use bins or when we filter records based on certain values, we may add misleading reference lines or create inaccurate charts – but that’s a topic for another day.

Understanding Customer Feedback: How Visualizations Quickly Guide Us to Useful Conclusions

Christine Muser — Fri, 06 Nov 2009 23:43:10 +0000

Have you ever had the “pleasure” of slogging through hundreds of feedback forms from a seminar or conference? Have you ever noticed how the mind seems to dwell on the negative comments, maybe even to the point that all the positives seem to loose their luster? That’s when crunching actual survey numbers can help put things into perspective: either there really were problems or we are about to fall prey to the naysayers and constant critics.

For example, this heat map makes it obvious that Session C was the least popular event in this conference – but notice also that the range of scores is fairly close together. In this example, attendees used a 5 point scale, with 1 being the least favorable score and 5 being the most favorable score. Seeing that the lowest average score was above 4 tells us that, overall, attendees were quite happy with this conference.

A heatmap showing how each individual session was rated along multiple criteria -- click on the picture to enlarge it

Of course we also want to understand what worked well and what didn’t work so well. Let us take a more detailed look at Session C.

Click the picture to enlarge it

Very quickly we can tell that the audience was critical of the session content and how it was presented rather than the speaker’s knowledge about the subject. More than 90% felt that the speaker had an adequate background to present on this topic. But less than 75% were happy with the way the information was presented and how it related to their job. A few people awarded low scores of 2 and 1, but it is encouraging that these low scores came from fewer than 10% of the audience members.

Just to provide a contrast, let’s also take a look at Session B, clearly the favorite of this event.

Individual Scores for Session B

There is no doubt that the audience rated this Session very highly across the board: 74% or more audience members awarded it the highest score across all measures. Notice that this session, too, received a handful of low scores.

As we can see here, sometimes we only need a few pictures to gain useful insights. No doubt, if this were a more involved customer survey, we would need more and very likely different graphs to discern the finer points of attitudes and perceptions — especially when such information is tracked over time. But pictures like these are a good start to find out where to focus such additional efforts.

Expanding our Visual Vocabulary

Christine Muser — Tue, 06 Oct 2009 00:03:14 +0000

Last week I wrote about the need to expand our visual vocabulary in line with software that allows us to graph complex data relationships and events in ever more meaningful ways. Let me expand on this point with an example.

Just about any introductory statistics course covers the draft lottery of the Vietnam War era as a case study for determining whether events occurred by chance or whether they were subject to some significant influence. Typically this case study involves a scatter plot and discussions about statistical significance, p-values and regression lines.

The section of the draft lottery we are discussing here involved the following process: Each Birthday was given a number from 1 to 366 (including leap day) so that 1 = Jan 1^st, 2 = Jan 2^nd, 3 = Jan 3^rd and so on until 366 = Dec 31^st. Each Birthday number was written on a piece of paper, put in a plastic capsule and then in a shoebox from which each was later drawn one at a time. The Draft Number represents the order in which each birthday was drawn from the shoebox.

The scatter plot below shows Birthdays vs. Draft Numbers along with a trend line to indicate the relationship between the two. The casual observer may not notice the relative sparsity of data points in the upper right hand quadrant, let alone grasp the significance of it. It takes careful observation to notice that a significant number of birthdays that occur later in the year have lower draft numbers and visa versa.

Scatter plot and trend line showing Birthday vs. Draft Number. P < 0.0001 - Click the picture to enlarge it.

During business discussions or presentations we often do not have time to explain statistical models and their implications. Sometimes we need to find more intuitive means to get our point across quickly and effectively. When constructed properly, visual displays can convey a lot of information very quickly.

To illustrate the results from the Draft Lottery more intuitively, I divided all birthdays into three equal groups: Group 1 includes days 1 – 122, Group 2 includes days 123 to 244 and Group 3 includes days 245 to 366. I then plotted the corresponding draft numbers in each Birthday Group.

At a glance one can see that the Birthdays tend to cluster toward the upper end of the Draft Numbers range in Group 1 and toward the lower end of the Draft Numbers range in Group 3. There is no need to talk about p-values and regression lines, just a quick note pointing out that Birthdays in the early part of the year had a better chance of receiving a high draft number than Birthdays that occurred later in the year.

Draft Numbers by Birthday Group. Click the picture to enlarge it.

An alternative representation may include reference lines for quartiles as in this illustration:

Draft Numbers by Birthday Group - including reference lines for quartiles. Click the picture to enlarge it.

Business decisions, of course, are based on more than fancy pictures and we need to be able to back things up through detailed analysis. But when time is short and we need to make our point quickly, the second graph helps us out much more than the scatter plot.

Acknowledgements:

A big “Thank You” goes to the authors of “Online Statistics: An Interactive Multimedia Course of Study” for providing the data and for the inspiration to visualize it in a format other than a scatter plot. This free online course was developed at Rice University, University of Houston, Clear Lake, and University of Houston, Downtown, with partial support from the National Science Foundation.

All graphs were generated using Tableau software.

Visual Analytics: What’s The Big Deal?

Christine Muser — Sat, 03 Oct 2009 03:32:33 +0000

During several conversations recently the following comment came up: “What’s the big deal with visual analytics? It’s just a bunch of pretty pictures!” It took a while, but it finally dawned on me that we have reached the Xerox-GUI-Macintosh stage for data analysis. The early versions of a graphical user interface (GUI) which were developed at what was then called Xerox-PARC were no commercial success until Macintosh – now better known as Apple – created a computer that allowed everyone to point and click rather than write arcane computer instructions.

While point-and-click is much easier than writing code, it still requires computer users to become familiar with what the computer can do and how to accomplish various tasks. It also requires standards about where to click and what should happen when certain actions are taken. All this knowledge and these standards had to develop over time and often through trial and error.

In some ways we have reached a similar stage for data analysis: visual analytics provides a new language through which non-analysts can explore and answer business questions. It frees the non-technical user from the analytic equivalent of writing code, that is, it frees them from the need to learn how to create graphics that – until now – required significant technical knowledge to generate.

As with any new technology, some mayhem ensues: we have to gain experience and learn through practice. We need to become fluent in the appropriate use of less familiar – yet oddly intuitive – graphing techniques like sparklines, heatmaps or small multiples. In short, we must develop a visual vocabulary beyond the bar charts and line graphs we know from Excel or PowerPoint.

And this brings me back to the Xerox-GUI-Macintosh comment from the beginning: as in those early days of learning the language of icons and point-and-click, we have now reached the point where more powerful ways of encoding data in a visual format is available to the lay person. Just as with the graphical user interface, we will some day look back and say “I can’t imagine a world without seeing data in pictures.” For at least a little while, those of us in the analyst professions need to act as interpreters and guides to those who are learning this new language. Sooner rather than later we will all get there.

Using Tableau to Picture Survey Data

Christine Muser — Mon, 31 Aug 2009 23:00:41 +0000

For several years I have participated in the Heartland Association of Research Professionals (HARP), a non-profit organization that provides educational and networking opportunities for nearly 200 professionals engaged in clinical research and related fields.

Recently we surveyed current and prospective HARP members to gather feedback regarding the services we provide. This gave me a nice opportunity to learn more about Tableau. While I did encounter some formatting issues, Tableau provided a quick and easy way to explore survey results.

Quite useful was the ability to set up all the graphs, tables and dashboards while the survey data was being collected. Periodically I could update everything simply by refreshing the underlying data. This allowed me to easily keep my fellow board members in the loop while data was rolling in. It also allowed me to send reminders to survey participants using actual completion times to assure them that participation would really take only take a few minutes of their time.

Questions that allow for multiple answers can be a bit tricky to analyze, but visualizing the results with Tableau made that analysis much easier. The example here illustrates the various research certifications held by HARP members. This question focused on research certification as opposed to certification in other domains.

Below is an example using graphical clues and annotation to illustrate which of four networking related services were used by current and prospective HARP members. I cannot imagine anyone who would want to read a written description of these results. A picture communicates this much better and without the proverbial “thousand words.”

To improve readability, Tableau provides a great variety of formatting and labeling options such as turning titles, column and row headings on or off.

Example 1: Title turned on, Row Headings turned off:

Example 2: Title turned off, Row Headings turned on:

From a communications standpoint, it would have been nice if there had been a way to include all options of the five point scale in this graph. The fact that no one selected “Somewhat Unimportant” nor “Unimportant” would be much easier to see if I did not have to talk about it in a comment.

Using nothing more than zip codes as geographic coding data, Tableau provided us with a quick idea about the geographic dispersion of our membership – very handy, since picking a good meeting location is always a tricky topic.