Is the data clean enough to give us meaning? If you are one of those who insist that the war on dirty data must be won first, forgive me. I’m tired of that conversation. It sounds like backyard mechanics comparing fuel injectors when all that really matters is the commute.
I talked recently to TDWI’s just-departed education director Dave Wells. He’s been thinking, and he says, “Tweaking the details probably doesn’t make any real, significant contribution to the overall value of the insights I can gain by understanding the patterns inherent in data.”
Is it really, really going to make any difference if you can get your data from 96 percent to 98 percent pure? Will the meaning really change?
My old boss in market research had a rule of thumb: with a good sample, results usually stabilize after about 25 percent of the data have been counted.
Are we there yet? Wells says the more interesting problem these days how to soak the meaning out of what you do have. Let those who scrub keep scrubbing while the rest of us move on.
To get to the next level in BI, he says we’ll have to understand better how people think. That could involve systems thinking, how one part of an operation affects a different one. Other areas he’s looking into are design thinking and critical thinking.
Notable quotes
- One of the real looming questions I think in BI as it becomes mainstream is what do you do when analytics contradict conventional wisdom. Then it becomes a political and religious debate.
- The flaw in our current approach to analyzing things is best described by an analogy that says, “you cannot understand the impact of Van Gogh’s Starry Night by categorizing the brush strokes.”
- Maybe the value of real high impact metrics, for instance, is not in the measuring of things, but in the measuring of relationships.
- There will always be something technological to fix. Let’s go back to Starry Night. Do I need to fix that brush stroke? Or do I need to stand back and look at the big picture?
One person I’ve talked to about his ideas thinks Dave is “full of crap.” He says, “If you talk to the people who deal with data, you hear that the problem is nowhere near solved.” OK, there’s still work to be done. But rejecting Dave’s reasoning is like saying that because car design is still not perfect we can’t yet talk about driving from San Francisco to Lake Tahoe.
The full interview will be at the May 7 issue of BI This Week.
Links he suggests, with his notes:
- Systems thinking
- Design thinking: The weblog Functioning Form’s “Defining Design Thinking” post gives a good starting point, he says, with links to articles and papers. He writes, “Design thinking is a less mature discipline [than systems thinking], which makes it less robust but more interesting to explore.”
- Critical thinking: “The critical thinking community is more focused on education than on business today,” he writes, “so much of the interesting stuff there is in discovering how it does apply in business. I have no doubt that it does apply because much of critical thinking is simply another perspective on decision theory.” See the Strategies for Success page and the Foundation for Critical Thinking site.
Leave a Reply