Define, Measure, Analyze, Improve, Control

Whether we are aware of it or not, we employ these five steps all the time.  In order to solve a problem we first have to understand it (“Define” and “Measure”), then we have to choose a solution (“Analyze” and “Improve”) and to make sure the solution sticks, we have to put some “Controls” in place.   Lean Six Sigma shortens these five steps to form the acronym DMAIC and organizes projects into five phases called Define, Measure, Analyze, Improve and Control.   Each phase deserves careful attention.   Faults in any of them will create problems down the road, either by solving the “wrong” problem, by implementing the “wrong” solution or by creating an atmosphere where the habits that created the problem can re-emerge.   Whether we have to manage a project or try to solve a less complex problem, DMAIC is a good place to start.

Lean Six Sigma Provides Tools and Techniques to Ensure Success

Good business decisions require relevant information and the ability to get it. Elsewhere in this blog I have discussed some of the finer points of relevant information.  Let me focus here on “… the ability to get it” because that often depends on the skills, knowledge and motivation of the humans who provide the information.

Throughout our certification course we spent considerable time sharing real life stories and discussing what it takes to build team consensus, to make team decisions and to prioritize solutions.  Lean Six Sigma provides a toolbox of methodologies from which the adept practitioner can choose the ones that fit the team dynamics and the problem at hand.  The mechanics of these tools are easily learned – the human element can be more difficult to manage.

It takes human judgment and input from people to determine which factors are relevant, to discover where the problems are and to identify which solutions are feasible and should be pursued.  Motivations such as job protection, maintaining a good reputation, demonstrating leadership and controlling one’s destiny are powerful factors that affect not only team dynamics but also what information people are willing to share.   Lean Six Sigma calls itself a “data driven” methodology, but that doesn’t mean it ignores human input.  When used appropriately and with skill, Lean Six Sigma tools help to transcend these human factors by approaching the problem from many different angles and by placing the emphasis on processes and problem solving rather than blaming people.

Lean Six Sigma Is Data Driven

Data and statistical analysis play a central role in Lean Six Sigma and go far beyond the measurement of technical specifications.   Our “gut” will often point us in a good direction, but to get funding and to understand whether and where we are making progress, we need some numbers.  That reliance on “numbers” is explicitly built into the Lean Six Sigma process by requiring us to “Define” our problem, to “Measure” the current state and then to “Analyze” it to determine the best solution.   Hypothesis testing, Chi Square tests, ANOVA, regression analysis, t-tests and a host of other statistical tools used in Lean Six Sigma also work away from the factory floor: they enable us to understand patient motivation, provider opinions, sales rep performance and driving forces in the market place – to name just a few.

Subject Matter Expertise Still Matters

Being able to use Lean Six Sigma jargon like “Cause & Effect Matrix”, “Design of Experiment” or “Value Stream Mapping” doesn’t mean much unless we provide the necessary context.  Usually this means dropping the jargon and applying relevant subject matter expertise.   A “Cause and Effect Matrix” may provide the foundation for translating business priorities into a bonus plan – complete with performance goals and payout curve.  Concepts from “Design of Experiment” apply to “Survey Design” in Marketing Research as well as to “Conjoint Analysis” when we are trying to understand the impact of various market forces.   Creating a “Value Stream Map” may help with restructuring departments and job descriptions to support growth for a provider of healthcare or other services.  It’s not the tool that matters, it’s how we use it.

The Take-Away

Whenever conversations with friends and colleagues turned from abstract to more details about Lean Six Sigma, I started to hear comments like “oh, you’re doing a mini-MBA” or “that’s what I learned as part of my PMP certification” or “hey, this is an idea I can use.”   The discussion above illustrates how these comments came about.  When choosing among consultants, shouldn’t we give priority to someone who has demonstrated their ability to solve problems effectively and efficiently?  I am banking on it, and together with some new insights from class, I now have more than a “piece of paper” to demonstrate that ability to anyone who needs to know

This statement leaves the unfortunate impression that decisions based on theoretical models are somehow suspect and of little value.  The better, but less soundbite-friendly, question should have been whether the model adequately predicts the risk. To make useful predictions, a model needs to do two things:

• account for all key factors that influence the outcome
• quantify how each factor influences the outcome.

To the frustration of passengers, airlines and businesses around the world, the eruption of the Eyjafjallajokull volcano in Iceland presented several challenges for modelers.  Some of the key factors playing a role here include:

• Ash composition: how damaging is it?
• Extent of the cloud: how high, how far and where?
• Aircraft capability: how much ash exposure can airplanes handle?
• The costs of staying grounded vs. the costs of catastrophic failure

These four bullets of course represent just the tip of the ice berg: each of them summarizes a long list of related factors.  For example, while current data is unavailable, scientists have to rely on past experience when estimating the size and weight of ash particles, their chemical make-up and the density of the ash.  To confound matters, predictions regarding wind speed, direction and turbulence need to be considered as well.

So the real question needs to be: how well does the model represent the current situation? How well do we understand what the actual key factors are?  How well can we measure them?  Are we relying purely on past insight or can we refine our knowledge with data from the current situation?  To what extent can current satellite images, air samples and meteorological measurements improve our ability to predict the risk to life and well-being of people and property?

This interactive map on the BBC we site shows one example of how output from such modeling looks by mapping the extent of the plume over several days along with normal flight routes across the Atlantic.  This post in Business Week sheds some light on our spotty knowledge regarding the real risks of volcanic ash.

The other critical question revolves around acceptable risk.  Risk not only originates from ash clouds. It also comes from “playing it safe.”  Billions of dollars in lost revenues and productivity put the livelihood of hundreds of thousands of people at risk.  Every day we accept the risks of driving our car – so at what point do ash clouds represent a higher risk than driving a car?  Even if we could come up with an exact numeric value for these risks, how does the value of human life fit into these equations?

Therein lies the apparent disconnect between statistical models and real life: intangible values sometimes outweigh what can be measured.  The decisions we make depend on how we actually perceive the risk.  Yet, in order to put our perceptions into perspective, we need to have good numbers to guide us – and getting good numbers requires a good model of reality.

So, before we start talking about law suits, we need to accept that risk is inherent to anything we do.  Blaming people for doing the best they can to balance public safety with economic considerations wastes resources that would be better spent on improving our ability to assess and manage risks.  We send unmanned drones to gather combat intelligence, why not modify them to collect air samples?  Why not fund research to create better models for volcanic plumes? Especially if history should repeat itself and Eyjafjallajokull continues to sputter for the next year or two.

Additional Reading

From Eruptions, a blog dedicated to volcanism:

Airlines lobby to reopen European airspace closed by Eyjafjallajökull
Posted on: April 18, 2010 2:30 PM, by Erik Klemetti

http://scienceblogs.com/eruptions/2010/04/airlines_lobby_to_reopen_europ.php

Eyjafjallajökull flight cancellations: How the right decision is being made to look wrong

Posted on: April 22, 2010 9:40 AM, by Erik Klemetti

http://scienceblogs.com/eruptions/2010/04/eyjafjallajokull_flight_cancel.php

Research Gap Left Airlines Exposed to Volcano’s Blast (Update1)

April 22, 2010, 9:05 AM EDT

http://www.businessweek.com/news/2010-04-22/research-gap-left-airlines-vulnerable-to-volcano-s-fire-and-ice.html

Is driving more dangerous than flying through ash?

Page last updated at 10:51 GMT, Wednesday, 21 April 2010 11:51 UK

http://news.bbc.co.uk/2/hi/uk_news/magazine/8633484.stm

Could aircraft dodge the volcanic ash cloud?

Page last updated at 14:06 GMT, Tuesday, 20 April 2010 15:06 UK

By Stephen Mulvey

BBC News

http://news.bbc.co.uk/2/hi/in_depth/8632583.stm

Recriminations erupt in ash-fueled aviation crisis

AP 4/21/2010

By Arthur Max, Associated Press Writer

http://news.yahoo.com/s/ap/eu_iceland_volcano

How volcanic ash could ground your next flight

By Larry Dignan | Apr 15, 2010

http://www.smartplanet.com/business/blog/smart-takes/how-volcanic-ash-could-ground-your-next-flight/5989/?tag=content;col1

How Volcanic Ash Can Kill An Airplane

Apr 15, 2010 09:00 AM

Ray Wert

Story & pictures of KLM Flight 867

http://jalopnik.com/5517775/how-volcanic-ash-can-kill-an-airplane

Iceland Volcano Vs Mt. St. Helens And Airspace

Pilot’s discussion forum

http://www.airliners.net/aviation-forums/general_aviation/read.main/4783270/

Volcanic Ash Contingency Plan

ICAO (International Civil Aviation Organization)

http://www.paris.icao.int/documents_open/files.php?subcategory_id=63

http://www.paris.icao.int/documents_open/show_file.php?id=274

Here is an example of how such an exploration may look: we are exploring data about obesity, soda consumption and sales taxes on soda.  We are told this data came from the US Department of Agriculture and a quick look reveals that we are looking at county level data.  As one might expect, a scatter plot reveals a strong relationship between rising soda consumption and increased obesity.

Adult obesity rates increase as soda consumption increases

Now we get to the real questions: do sales taxes on soda help with lowering obesity rates?  What relationship do we see between sales tax rates on soda and obesity? As luck would have it, the data we received also provides two measures about sales taxes for soda: one rate for vending machines and another rate for retail stores.

First we look at the relationship between soda taxes for retail stores versus obesity rates.  One might expect that taxes discourage soda consumption and, yes, there appears to be a small downward trend as tax rates increase.  Maybe soda taxes actually help with bringing down obesity?

Adult Obesity Rates, Retail Sales Tax Rates and Soda Consumption by US County

Now let’s take a look at sales taxes on soda coming from vending machines.  Interesting observation: diabetes rates seem to increase slightly as these tax rates increase.  Counter intuitive?  How do vending machine purchases differ from purchases in a retail store?  Are we observing a real relationship here, or is the data fooling us?

Adult Obesity Rates, Vending Machine Sales Tax Rates and Soda Consumption by US County

Before answering these questions, let’s take a closer look at all those data points on the y-axis.  Do they really indicate that these counties levy a 0% soda tax?  A quick inspection of the underlying data shows that, yes indeed, all records indicate a 0% tax rate.  Not a single “null” value among them.   However, without knowing how the data was processed, we cannot be sure that “zero” really means “no taxes” – it could also mean “no data.”

To explore further we start by placing the three graphs side by side.  This way we can see more easily what happens when we exclude “zeroes.”

Soft Drink Consumption, Obesity and Soda Sales Taxes

First we exclude “zeroes” for retail sales taxes. Then we’ll do the same with taxes levied on soda in vending machines.  The following graphs illustrate this.

Excluding Records with 0% Soda Sales Tax Rate (Retail). The center graph shows the relationship between the remaining retail records and obesity. The trend line still points downward.

Excluding Records with 0% Soda Sales Tax Rate (Vending). The right hand graph shows the relationship between the remaining vending machine records and obesity. We now see a downward trend as tax rates increase.

Wait a minute, though.  When we exclude “zeroes” from one set of taxes, all data points for “greater than 0% taxes” disappear from the other graph.  In other words, this data indicates that the two types of taxes are mutually exclusive!  Hmm, does this even make sense in real life?  Why would every US county tax soda either in retail stores or in vending machines but never in both?

Without further knowledge about this data we have to reframe our questions and conclusions:

• When soda taxes are levied, higher tax rates appear to go hand in hand with decreasing obesity rates
• We cannot draw any conclusions about the impact of “no sales taxes” versus “sales taxes”
• Before we continue with a detailed analysis, we probably need to ask questions about this data.  At first glance it makes little sense that counties levy soda taxes either on vending machines or on retail stores but never on both.  Then again, I’m not a tax expert.

Chances are that we will uncover other areas about which we need to ask questions.  Instead of taking the scattershot approach to learning about this data, data exploration helps us to develop very specific questions to ask.  With specific questions, we stand a better chance of finding the right subject matter experts to consult.

This was a quick example for exploring data about which we knew nothing when we started.  To gain new insights, we sometimes need to apply this “beginners mind” approach even to data about which we already know a lot.  After all, errors can happen, collection and processing systems can change without our knowledge and sometimes we find nuggets that were hidden until we started looking for them.  One final thought: the next time your boss or client asks to hurry up with the analysis, ask these two questions:

• What are the consequences of making poor decisions because we hurried too quickly through the data exploration?
• Do we need to go for more accuracy or is a ballpark analysis good enough at this time?

• How does one differentiate between the “voice of the customer” and the “noise of the customer?”

VOC or “Voice of the Customer” is a key concept in Six Sigma, the quality methodology used by Toyota and many other companies.  Needless to say that with millions of customers, there are millions of opportunities for feedback – hence the potential for noise.

Wordplay aside, any communication from a customer contains some useful information, but not all feedback carries the same weight.  For example, a broken radio most likely has less impact on car safety than a stuck gas pedal – but we can’t be sure until we have more information: the broken radio may be a symptom of an electrical problem that also affects the accelerator.

Therein lies the problem: how do we assign the “appropriate” value to the information we receive?  How much effort and money do we put into researching the (hypothetical) “radio problem” versus other problems?  How can we quickly assess whether the “radio problem” can turn into a “safety problem” that requires thorough attention?  With the myriad of active and passive ways in which we can listen to customers, we need a good triaging system to help us separate critical information from information clutter.

While everyone can agree that data needs to be used “appropriately,” it is much more difficult to agree on what “appropriate use” actually means.  Assuming for the moment that we can collect accurate data, what do we need to know in order to elevate an incident from “routine” to “requires immediate attention?” Here are several key factors that influence appropriate use:

• The ability to recognize the potential for significant harm
• The ability to draw a correlation between the incident and significant harm
• The ability to develop a solution to the problem
• The ability to implement a solution to the problem
• The ability to make that solution pay off in the long run

Each of these bullet points shares two characteristics: to accomplish them, we need good information as well as sound judgment – neither of which comes easily.  This applies to all types of corporate decisions – whether we are dealing with product safety issues or the most profitable allocation of sales and marketing resources.  The major differences between types of decisions typically revolve around their scale and the level of detail required to make a decision.

It is impractical to go through all the possible ways in which we can identify “appropriate” information.  Instead, here are a few guidelines:

• Assess the potential harm
• Identify actionable information
• Prioritize timeliness, accuracy and budget
• Identify who needs to know what and when
• Incorporate the means to review requirements from time to time

Keeping these bullets in mind goes a long way toward selecting the tools and resources needed to supply appropriate information.

Additional Reading

Toyota knew of accelerator pedal problem in UK a year ago
From The Times
February 2, 2010

http://business.timesonline.co.uk/tol/business/industry_sectors/transport/article7011671.ece

Unintended Acceleration: Toyota Addresses the Issues
November 06, 2009 by Irv Miller

http://pressroom.toyota.com/pr/tms/our-point-of-view-post.aspx?id=2234

Wikipedia entry for Six Sigma, the quality control methodology used by Toyota and many other companies.  Voice of the Customer (VOC) is a key concept of the Six Sigma methodology.