There are the dream organizations that deploy data analysts wisely. Then there are the nightmares, such as the I.R.S. as portrayed in David Foster Wallace’s last novel, The Pale King, reviewed yesterday in the New York Times.

… In a universe of veiled and veiling numbers, the task of drawing the true [data] out into the light and holding them up for inspection, clear and remainder-­less, really is a sacred one. … The problem, as I.R.S. recruits soon discover, is that neither moral nor heroic codes hold true anymore.

These recruits work with “excruciating difficulty … in an age of data saturation.”

The [instructor] presents “the world and reality as already essentially penetrated and formed, the real world’s constituent info generated . . . now a meaningful choice lay in herding, corralling and organizing that torrential flow of info.”

One character is the data psychic, Sylvanshine, who can “glean trivia about anyone simply by looking at him.” But, as if to prove that good data is far from the end of the story, he has a problem.

[He] is “weak or defective in the area of will.” Nor, due to endless digressions, can he complete anything. No one can; in “The Pale King,” nothing ever fully happens. That this is to a large extent a metaphor … becomes glaringly obvious when we hear one unnamed character describe the play he’s writing, in which a character sits at a desk, doing nothing; after the audience has left, he will do something — what that “something” is, though, the play’s author hasn’t worked out yet.

Let’s see, will an “easy to use,” “speed of thought” tool help? Is there a tool for Sylvanshine and the others?

No, at least not until the next update. But this is why business intelligence is fascinating. Under cover of tools and data, we touch the heart — throbbing or dead — of the organization.

What will it be, a “single version of the truth” or unabated proliferation of ad hoc data? It’s a chronic dilemma, and its resolution is crucial to big-box business intelligence. Frank Buytendijk’s new book, his second one, offers a way out of this pickle.

In Dealing with Dilemmas: Where Business Analytics Fall Short (John Wiley & Sons; 2010), Buytendijk — pronounced BOW-ten-dek, according to a Dutch friend of mine — argues that the usual this-or-that, you-or-me, and now-or-then dilemmas may not be the tough choices that they seem to be.

I had known Frank Buytendijk from his two TDWI keynotes, both of which broke down old fences. Then I got to the part of the book where he takes on Michael Porter — author of the essay “What is Strategy?,” in which he defines strategy partly by what a business doesn’t do. Southwest Airlines, for one well known example, offers no reserved seats or meals. But without that distraction, it can fly you on time at a reasonable price.

Porter’s theory, I used to say, is comparable to cropping a photo: emphasize one aspect by trimming others. Then you know what the message is for once and for all, or maybe you have to interpret it, but what you need is all there. Buytendijk’s theory may be more like making a movie. The movie, too, requires the artist to decide on emphasis and exclusion, but in a movie the story plays out over time and through multiple spaces. The movie, too, has a message. But a movie goes this way and that way as it winds toward the end — like a business as it winds through its environment toward a goal.

Of course, a movie requires the artist to think harder. A movie takes shape much more slowly. I’ve only imagined Buytendijk’s principles in practice, but I think that what he prescribes is far more involved the usual strategy formulation.

Buytendijk, in fact, has fun ridiculing strategy-formation executive campouts. Inspiration may strike while they “sing songs around the campfire.” The marshmallows and scotch taste good, but the thinking doesn’t stick, the assumptions are forgotten, and the organization is left to live on slogans.

What’s required, he writes, is deeper understanding of the theory behind the business and the nature of the dilemmas that decision makers face.

While the goal (that we chose) remains intact, and the assumptions remain in place as long as they match reality, we can travel toward our goal, assessing whether options that we create and opportunities that we see fit into the framework. If so, we capitalize on them; if not, we let them go. And the moment assumptions change, we can immediately see which activities do not lead us to the goal anymore, or which activities are lacking in making it to the goal. Choices do not turn into dilemmas.

Choices don’t turn into dilemmas the way “single version of the truth” versus spreadmarts has. The Big Brother version of decision support might have been devised in a campfire sing-along — far away from those who still had work to do.

What would Buytendijk do with that problem? I think he would classify it as a “you-or-me” type. It involves one group against another, usually IT soldiers charged with enforcing a policy against rebel cells armed with spreadmarts. But if either one had a decisive victory, it might spell trouble for the organization.

He prescribes three steps: First, examine your motives. What are you really trying to do? What’s the goal? And so on. Next, communicate. Do not fall for that old slogan of ham-fisted managers, “If you’re not part of the solution, you’re part of the problem.” The solution may be found in conversation. “By being part of the solution from the start,” he writes, “the only angle you will see is your own … Acknowledging there are multiple sides to the story, even if you do not agree, is the key to reconciliation.” Finally, reconcile and synthesize. Opposites — such as love and hate, Tea Bagger and Berkeley liberal, IT soldier and spreadmart rebel — may actually be more alike than you think.

I see what he means when, at the end, he confesses to not understanding the old saying that “you can’t have your cake and eat it too.” It’s stupid, something a burned out school teacher uses to keep order. Aren’t we smarter than that? And if there’s hope for cake, there’s hope that the “single version” and the spreadmarts can live together under one roof.

Business is ultimately not technical but social, is it not? Appropriately, this book is deeply humane and intelligent — expressed in a warm, conversational voice. That alone distinguishes it from most other business books. It eases you through difficult new ways of thinking, through what I think many readers will find is new and unfamiliar territory. Decision makers who are willing to put in the effort to understand it and put it into practice, I think, will find it worthwhile.

Dealing with Dilemmas may never become a mainline classic of the kind Porter wrote. But it will certainly be a favorite among a smart and adventurous few.

What can explain business intelligence’s poor adoption rate? Are tools not easy to use? Or is there a deeper reason?

A book from 2000, The Social Life of Information by John Seely Brown and Paul Duguid, suggests that BI designers have neglected basic human needs. Jack Vinson, of Knowledge Jolt with Jack fame, has just posted a worthwhile review that sent me scurrying over to Amazon.

Failure begins early for many new, supposedly revolutionary information systems. Designers “assume that the way people operate with respect to information has to do with only the information. … But there is a social life that revolves around the information that is much harder to capture and codify,” Vinson writes. “We look to verbal and physical queues for validity of what someone is saying. Our business processes have much more than just the inputs and outputs.”

Jumping forward but on the same thread:

… in the essay on reengineering … the authors describe how all the social life around business process is downplayed and often treated as waste. Businesses were re-engineered to remove much of the social lubricant that helped business flow. The essay on knowledge management was hopeful that KM would be a shift away from the intense focus on information and account for the human aspects of knowledge: that knowledge requires a knower. They have a great phrasing: information can easily be written down and transferred. But it is much harder to detach (and transfer) knowledge from the know-er and the context in which that knowledge resides.

The book is still important even after 10 years. It doesn’t even mention business intelligence, yet it addresses some of its fundamental problems.

Take a look at The Social Life of Information on Google Books. I also recommend Knowledge Jolt with Jack. Always worthwhile.

Back when secretaries were common, you could have had yours track your day in 15-minute increments. In his book The Effective Executive, Peter Drucker suggested this as a way to find out what you really did all day. The picture usually wasn’t so pretty.

Tracking your time then and now is personal, it’s messy, and it’s the essence of business intelligence: collecting data and reading it for guidance in business activities that matter. Is there anything that matters more to an organization than productivity of its people? For a small office or home-based business, this might be the best BI there is.

This gets no recognition in the BI industry that I can find, at least not in the conservative world of TDWI. At least not yet.

PI — for “private intelligence” — has different issues, starting with data collection. In BI, data comes from transactions, all recorded routinely. In PI, most of it has to come from a “secretary” or from our own, tedious notation.

I dabbled in it once. The insights were good, if painful, but mostly it was tedious. A few years ago, a confluence of personal events let me do something I’d always wanted to try: hole up for a few months in a Sicilian village I knew slightly. The food was good, I had relatives nearby, and the nearby church bells rang all day and all night, four times an hour. At the same time, I had a book to edit. To stay productive, I made a game out of the work, tracking my time to the minute in Filemaker.

I liked the local food and started to hate the book, an office manual that inadvertently revealed a con game. Even so, I threw myself at it every day. But no matter how hard I tried, no full day ever resulted in more than about two hours of actual, productive work. My “quick breaks” for walks and coffee with a friend actually took up more time.

I made a Filemaker database because I could find no off-the-shelf product that would do anything close. Each period, no matter how short, had a starting and ending times I entered with buttons, and a calculation field figured the duration. A drop-down menu offered my usual activites. I could make a report for any period.

I thought some product would do that better, but I could find nothing. Then the May 2 issue of the New York Times Magazine ran an article by Gary Wolf about this, “The Data Driven Life.” My Filemaker invention wasn’t too far from what others have used, and now new devices are coming along that could make all that seem so old hat. Some people are even sharing their data on the cloud.

But as in traditional BI, the technology just gets you in the door. The show has just begun.

Most people Wolf writes about do it for personal reasons. One wanted to know how his coffee consumption helped him focus, another tried to cure his sleep apnea, and still another noticed that flax seed oil, or just lots of butter, improved his cognitive performance.

As in good BI, the experiments often raised new questions. And sometimes the new questions are unexpected, as in Wolf’s own experience.

Often, pioneering trackers struggle with feelings of being both aided and tormented by the very systems they have built. I know what this is like. I used to track my work hours, and it was a miserable process. With my spreadsheet, I inadvertently transformed myself into the mean-spirited, small-minded boss I imagined I was escaping through self- employment. Taking advantage of the explosion of self-tracking services available on the Web, I started analyzing my workday at a finer level. Every time I moved to a new activity — picked up the phone, opened a Web browser, answered e-mail — I made a couple of clicks with my mouse, which recorded the change. After a few weeks I looked at the data and marveled. My day was a patchwork of distraction, interspersed with valuable, but too rare, periods of focus. In total, the amount of uninterrupted close attention I was able to muster in a given workday was less than three hours. After I got over the humiliation, I came to see how valuable this knowledge was. The efficiency lesson was that I could gain significant benefit by extending my day at my desk by only a few minutes, as long as these minutes were well spent. But a greater lesson was that by tracking hours at my desk I was making an unnecessary concession to a worthless stereotype. Does anybody really believe that long hours at a desk are a vocational ideal? I got nothing from my tracking system until I used it as a source of critical perspective, not on my performance but on my assumptions about what was important to track.

I wish Drucker were around to respond. Wolf’s insight sounds like important stuff for everyday knowledge workers, especially those who work alone. What’s more important to a knowledge worker than time?

These experiments are often haphazard and highly personal.

Generally, when we try to change, we simply thrash about: we improvise, guess, forget our results or change the conditions without even noticing the results. Errors are possible in self-tracking and self-experiment, of course. It is easy to mistake a transient effect for a permanent one, or miss some hidden factor that is influencing your data and confounding your conclusions. But once you start gathering data, recording the dates, toggling the conditions back and forth while keeping careful records of the outcome, you gain a tremendous advantage over the normal human practice of making no valid effort whatsoever.

Yes, just as analytics gives companies a tremendous advantage over those who make less effort.

Let the BI traditionalists pooh-pooh self-tracking. The very same people might have dismissed such things as visual analysis, agile development, and at one time even business intelligence itself. Sometimes it take a few pioneers and geeks, perhaps even a secretary, to prove a concept.

Stephen Few gave a snappy name to his new book, Now You See It, and a cover that signals a gem — all black with a slice of sunset that highlights the “see.”

Inside, many charts are so beautiful — at least to a visual analysis fan — that they rival the waterfalls and trees of old Sierra Club coffee table books. It’s on paper so thick — and even smoother than Few’s first book, Show Me the Numbers — that you might feel like you’re flipping postcards.

While Show Me the Numbers was about how to present visualized data, he explains, Now You See It: Simple Visualization Techniques
for Quantitative Analysis is about understanding them, which apparently includes how to make them.

Few’s a natural teacher. He reviews, again, the basis of visual analysis and how the brain is better wired for visualization than for row-and-column thinking. In each chapter, he takes the reader through by the hand from the most basic concepts of visual data analysis through complex ones. Toward the end, he points the student to the horizon to imagine a future state of visualization. I especially liked the interesting chapter “Prerequisites for Enlightening Analysis.”

My question after a while, though, was who Few’s “you” is. What kind of student does he imagine? If the student really requires the kind of patient explanation found throughout the book, shouldn’t that student do some other kind of work instead? Besides, isn’t visualized data supposed to be understood easily?

I’m a mere beginner at visual data analysis, comfortable with Excel charts but not fully competent with Tableau. Even so, the out-of-the-box brain function that visualization scholars call “pre-attentive cognition” seems to have guided me well through visualized data so far.

I can’t help but skim past some parts of the book. In the chapter on part-to-whole and ranking analysis, for example, he spends four luxurious pages on what I thought would have been too basic for adults. That includes nearly one whole page on the meaning of trend lines: some go up, some go down, and some stay the same.

Do these basic explanations bore me because I’ve seen at least parts of Few’s basic presentation too many times? Or did I learn enough from Show Me the Numbers, from Edward Tufte’s books and presentations, and from a stint with a Tufte-admiring market research boss?

Or could it be that, like so many other kinds of work, visual analysis is a lot easier to observe than to practice? Would I appreciate the explanations if I were up to my elbows every day in visual analysis? Hard to know.

Too basic for some, but the book could be just right for others. This book could be the bridge from the early adopters to mass adoption. Also, longtime analysts could appreciate the refresher — for that slap-on-your-own-forehead moment when reminded of basics.

Few conducts the book like a classroom, narrating with statements like, “We’ll look at the following techniques and best practices” as if coaxing freshmen through a long, hot afternoon session.

He does best when he illustrates points with a story. The chapter on distribution analysis begins with Stephen Jay Gould — diagnosed with a kind of cancer with a median mortality of only eight months after discovery. Gould thought about that statistic and discovered that he was a prime candidate for longer survival. He went on to live another 20 years, and helped this chapter seem shorter.

But there’s got to be more to visual analysis than the simple charts in this book. I wish Now You See It skewed toward more sophisticated visual analysis, the kind I hear about but have not yet reached.

In fact, the phrase “now you see it” has another half, ignored here: “now you don’t.” The two halves suggest a cycle of knowing and then asking again — what Tableau calls the cycle of visual analysis. The Tableau white paper “Visual Analysis for Everyone: Understanding Data Exploration and Visualization” describes it — visually, of course — on page five. I suspect the subject deserves a book.

Let’s see if Few’s got that one up his sleeve for later.

Imagine someone writing a book about data analysis without even mentioning software.

“To penetrate the heart and soul of the problem of improving analysis,” writes Richard J. Heuer Jr. in Psychology of Intelligence Analysis, “it is necessary to better understand, influence, and guide the mental processes of analysts themselves.” It’s the mind that does the heavy lifting.

Though Heuer writes for CIA-type analysts, similarities to business are clear. The biggest one is that it’s always the mind that does the heavy lifting.

The two kinds of data analysts share many other aspects of their work, such as their unconscious biases. For example, they all tend to give undue importance to vivid evidence over abstract evidence.

Both also endure the after-the-fact know-it-all, the type that announces, “That analysis told me nothing new,” after confirming evidence arises. Heuer describes an experiment that suggests such beliefs are usually delusional.

Analysts’ memories, too, undermine their work, such as when asked to remember what led them to their conclusions. Also, overseers tend to regard events as more foreseeable than they actually were.

Missing data, another problem, is “out of sight, out of mind.” Two groups of experienced auto mechanics were asked for a diagnosis of a car that wouldn’t start. They were offered a fault tree with several major branches — weak battery, starter-system defects, etc. — with several themes left out and summarized as “other causes.” One group’s fault tree included all major branches. But the second group’s fault tree had several branches missing — and selected the summarized “other causes” far less often than would be expected if the summarized reasons had been listed along with all the others.

Heuer offers a few solutions. The main one seems to be his “Analysis of Competing Hypotheses.” He outlines it in eight steps.

Heuer writes the way I’d expect a CIA veteran to write: formally. Even where he has a colorful tidbit — such as any of the load of stories mentioned in the footnotes — he opts for gray.

Such drab phrasing ignores the principle he gives in Chapter 10 under “the vividness criterion.” More vividness would have made a more readable book.

The impact of information on the human mind is only imperfectly related to its true value as evidence. Specifically, information that is vivid, concrete, and personal has a greater impact on our thinking than pallid, abstract information that may actually have substantially greater value as evidence.

If there’s one thing the book reminds the reader of over and over, it’s that analysts are only human.

On tools, Heuer lists only what he considers the basics: decomposition and externalization. Vendors will have to do what they can with those clues. Unlike so much software, though, Psychology of Intelligence Analysis is free.