Most people shopping for business intelligence tools think they’re sophisticated, observes a sales person who often represents a large vendor in TDWI exhibit halls. Most of these people are deluded.

He may sound harsh, but he observes all this not bitterly but with good humor.

“They buy [BI] like anything else they buy,” he explains. “They put out a bunch of RFPs and go through this whole process and stuff, and eventually they buy the one that’s maroon.”

Over years, he has identified the top three reasons people buy, and the underlying motivation:

1. Career. The chosen tool decides the career of those who will learn it, massage it, and become co-dependent with it. Greed guides the choice; a good choice may ensure steady, lucrative employment for decades to come.

2. The boss’s choice. The boss says, “Buy that one.” Buyers fear defying the boss and like the boss’s power to give status. The blessed few may seem sexy, at least to themselves.

3. Default. They buy one brand and only that brand. They say, “We’re an Oracle shop” or “We’re an IBM shop,” and for them that’s reason enough. These irk this sales person.

The brand loyalists are the ones who seem to irk him the most, even though his brand often benefits from their foolishness.

Even the brand’s high cost can’t shake their trust. “A lot of people assume that more money means more quality and less risk. No, it doesn’t!” he says.

“They buy this big, fancy thing that no one uses.” Instead, many would be better off with a less expensive brand that delivers almost all of the name brand’s features. The left over funds should be spent on training.

The reason many shy away from training? They’d have to admit that there’s ramp-up time. “It’s kind of the Sarah Palin view: ‘Do we really need sophistication?’”

There are the dream organizations that deploy data analysts wisely. Then there are the nightmares, such as the I.R.S. as portrayed in David Foster Wallace’s last novel, The Pale King, reviewed yesterday in the New York Times.

… In a universe of veiled and veiling numbers, the task of drawing the true [data] out into the light and holding them up for inspection, clear and remainder-­less, really is a sacred one. … The problem, as I.R.S. recruits soon discover, is that neither moral nor heroic codes hold true anymore.

These recruits work with “excruciating difficulty … in an age of data saturation.”

The [instructor] presents “the world and reality as already essentially penetrated and formed, the real world’s constituent info generated . . . now a meaningful choice lay in herding, corralling and organizing that torrential flow of info.”

One character is the data psychic, Sylvanshine, who can “glean trivia about anyone simply by looking at him.” But, as if to prove that good data is far from the end of the story, he has a problem.

[He] is “weak or defective in the area of will.” Nor, due to endless digressions, can he complete anything. No one can; in “The Pale King,” nothing ever fully happens. That this is to a large extent a metaphor … becomes glaringly obvious when we hear one unnamed character describe the play he’s writing, in which a character sits at a desk, doing nothing; after the audience has left, he will do something — what that “something” is, though, the play’s author hasn’t worked out yet.

Let’s see, will an “easy to use,” “speed of thought” tool help? Is there a tool for Sylvanshine and the others?

No, at least not until the next update. But this is why business intelligence is fascinating. Under cover of tools and data, we touch the heart — throbbing or dead — of the organization.

Haul away the hardware, peel off the software, rinse off the mystique and you see what the people who manage data really are: They’re librarians. That’s the role IT workers should model themselves on.

I’m not talking about technology. I don’t care what tools anyone uses. Whether we’re talking about bound paper known as “books” or bits magically transmitted over “wi-fi,” I don’t care. It doesn’t matter.

I know, the comparison may seem harsh. Librarians are said to shuffle silently among musty old books that no one ever reads. Or, as my friend Karen Schneider puts it, they’re “some misguided brontosaurus snuffling in the antediluvian biblioforest.”

She’s director of the Cushing Library at Holy Names University, just across the bay from San Francisco. She’s one of the actual librarians who resist a trend among some in her profession. They want to run libraries like traditional information technology departments. They’ve been seduced by the old mystique — which in the business world has worn thin.

You know the complaints: IT guards its data like gold bullion instead of serving it to those who can create value with it. It tries to shop its way out of problems. Only the initiated may enter.

Why anyone would want to emulate that, I don’t know. Yet apparently, from what she wrote last week in her blog Free Range LIbrarian, this trend has legs among some who manage libraries.

That trend seems idiotic when you realize what a well run library is all about. Substituting just a few words, you can see a philosophy for IT in the one she describes for librarians:

In the end, what matters, and what we are about, are the ancient truths of librarianship: organizing, managing, making available, preserving, and celebrating the word [data] in all of its manifestations; helping our users build skill sets the fundamentals of which (if not the ephemeral details) will last a lifetime [a fiscal year]; and celebrating and defending the right to read [analyze], however that word is interpreted. This is what we do. This is who we are. This makes us librarians.

Librarians and IT workers, that is. Does technology really make anything new? I say that, fundamentally, nothing is new but the tools.

When, when will someone write a novel about business intelligence? I can see it now. Amazon will try to sell me From Here to Analysis, Data in the Afternoon, and Lolita, DBA.

But titles are easy. Writing the novel might be tough, since the author would have to run on sheer imagination. There’s no apparent intrigue. The BI crowd plays it safe. Forget sex. The BI crowd works too hard. Forget guns. The BI crowd shoots only bullet points.

In fact, the best strategy might be Alfred Hitchcock’s favorite: the macguffin. It’s a plot element, often ephemeral, that drives the main characters to do what they do regardless of the macguffin’s value. As the plot moves along, the macguffin fades into the background.

In BI, the most common mcguffin is technology — while the real issue is effective use of data in business decisions.

Remember “Psycho”? The woman steals money from her boss and flees. Her boss is sure to follow. We’re looking off in that direction when out of nowhere comes the shower scene. Holy crap! We never saw that one coming!

Hitchcock’s aim was fear, but McGuffins have other uses, too. In business intelligence, the macguffin creates comfort. In the BI plot, the shower scene is instead endless talk about technology, data quality, data this and data that. All the time, lurking in the culture, unexamined and feared, is classic business dysfunction. But leadership can’t fix it, so they go shopping.

What’s a novelist to do? Use it all. Study up on the technology for a week and interview people on the front lines. Then drape BI technology over a standard plot set in an enterprise.

Business people buy BI tools, and so they’ll buy the BI thriller.

What drove 39 tweeters to endorse CIO Insight’s latest monument to redundancy? I wonder how many actually read it all. I could hardly reach the first period before fatigue set in.

The blog post titled “Gartner: CIO as Business Transformation Leader,” dated November 1 and promoted on CIO Insight’s email blast last week, begins with an admonishment to CIOs: “it’s time” that they plan to emerge as a “change leader.” OK, but shouldn’t any CIO know that already?

Ten questions follow, including “What type of change is happening?,” “What is the deadline for the change?,” and “What are the constraints …?” followed by explanation and the most basic of advice. Anyone but undergraduate business students should have heard it all before.

It all takes up 1,312 words when the same ideas could have been expressed in half that. What makes people think this stuff is worth a tweet?

What will it be, a “single version of the truth” or unabated proliferation of ad hoc data? It’s a chronic dilemma, and its resolution is crucial to big-box business intelligence. Frank Buytendijk’s new book, his second one, offers a way out of this pickle.

In Dealing with Dilemmas: Where Business Analytics Fall Short (John Wiley & Sons; 2010), Buytendijk — pronounced BOW-ten-dek, according to a Dutch friend of mine — argues that the usual this-or-that, you-or-me, and now-or-then dilemmas may not be the tough choices that they seem to be.

I had known Frank Buytendijk from his two TDWI keynotes, both of which broke down old fences. Then I got to the part of the book where he takes on Michael Porter — author of the essay “What is Strategy?,” in which he defines strategy partly by what a business doesn’t do. Southwest Airlines, for one well known example, offers no reserved seats or meals. But without that distraction, it can fly you on time at a reasonable price.

Porter’s theory, I used to say, is comparable to cropping a photo: emphasize one aspect by trimming others. Then you know what the message is for once and for all, or maybe you have to interpret it, but what you need is all there. Buytendijk’s theory may be more like making a movie. The movie, too, requires the artist to decide on emphasis and exclusion, but in a movie the story plays out over time and through multiple spaces. The movie, too, has a message. But a movie goes this way and that way as it winds toward the end — like a business as it winds through its environment toward a goal.

Of course, a movie requires the artist to think harder. A movie takes shape much more slowly. I’ve only imagined Buytendijk’s principles in practice, but I think that what he prescribes is far more involved the usual strategy formulation.

Buytendijk, in fact, has fun ridiculing strategy-formation executive campouts. Inspiration may strike while they “sing songs around the campfire.” The marshmallows and scotch taste good, but the thinking doesn’t stick, the assumptions are forgotten, and the organization is left to live on slogans.

What’s required, he writes, is deeper understanding of the theory behind the business and the nature of the dilemmas that decision makers face.

While the goal (that we chose) remains intact, and the assumptions remain in place as long as they match reality, we can travel toward our goal, assessing whether options that we create and opportunities that we see fit into the framework. If so, we capitalize on them; if not, we let them go. And the moment assumptions change, we can immediately see which activities do not lead us to the goal anymore, or which activities are lacking in making it to the goal. Choices do not turn into dilemmas.

Choices don’t turn into dilemmas the way “single version of the truth” versus spreadmarts has. The Big Brother version of decision support might have been devised in a campfire sing-along — far away from those who still had work to do.

What would Buytendijk do with that problem? I think he would classify it as a “you-or-me” type. It involves one group against another, usually IT soldiers charged with enforcing a policy against rebel cells armed with spreadmarts. But if either one had a decisive victory, it might spell trouble for the organization.

He prescribes three steps: First, examine your motives. What are you really trying to do? What’s the goal? And so on. Next, communicate. Do not fall for that old slogan of ham-fisted managers, “If you’re not part of the solution, you’re part of the problem.” The solution may be found in conversation. “By being part of the solution from the start,” he writes, “the only angle you will see is your own … Acknowledging there are multiple sides to the story, even if you do not agree, is the key to reconciliation.” Finally, reconcile and synthesize. Opposites — such as love and hate, Tea Bagger and Berkeley liberal, IT soldier and spreadmart rebel — may actually be more alike than you think.

I see what he means when, at the end, he confesses to not understanding the old saying that “you can’t have your cake and eat it too.” It’s stupid, something a burned out school teacher uses to keep order. Aren’t we smarter than that? And if there’s hope for cake, there’s hope that the “single version” and the spreadmarts can live together under one roof.

Business is ultimately not technical but social, is it not? Appropriately, this book is deeply humane and intelligent — expressed in a warm, conversational voice. That alone distinguishes it from most other business books. It eases you through difficult new ways of thinking, through what I think many readers will find is new and unfamiliar territory. Decision makers who are willing to put in the effort to understand it and put it into practice, I think, will find it worthwhile.

Dealing with Dilemmas may never become a mainline classic of the kind Porter wrote. But it will certainly be a favorite among a smart and adventurous few.

A young information technology worker in a large organization follows his common sense — and his boss scolds for it. The question is how to respond.

Back in 1980, he had just started at his first job, at the CBS Television Network. He soon noticed that every week business people asked him for the same data from the same source. So he did the logical thing: he wrote an algorithm for himself to save time.

Then he did the next logical thing: he gave the algorithm to the business people. They could do it themselves, and he didn’t have to do it for them. Everyone was happy.

That is, everyone was happy until the boss heard. The young man was called into the office.

“You gave them an algorithm?,” the boss asked.

“Yes, sir.”

“So now they can extract the data all by themselves?”

“Sure. They always ask for the same thing, so I thought they’d like it better if they didn’t have to ask me.”

“Lou, if they can do that, what’s our job?”

His ultimate response is a good product — which he won’t let me identify — that gives users control of their data.

Many in the business intelligence industry talk about organizational problems getting in BI’s way, but few talk about them very much.

Scratch the surface of most presentations and conversations — such as last week at the TDWI conference in San Diego — and you find people problems bobbing right up alongside data problems: indifferent executives who undermine BI, short-sighted silo keepers, and IT people who enrage business users with paternalism, to name a few top quirks. If only data were all we had to transform!

One business manger at last week’s TDWI conference in San Diego told me that one of his most daunting tasks during a recent data warehouse implementation was persuading silo managers to release their death grip. For this task, he was on his own. Couldn’t someone have briefed him on the objections he was likely to hear? Or tactics to overcome resistance?

One organization that seems to have solved its people-problem was Blue Cross Blue Shield of Kansas City. Their impressive success with Hewlett-Packard tools was based on commitment to data for strategic advantage and shrewd orchestration. They also had a steady, guiding hand from HP. For example, as Blue Cross Blue Shield built new structures, it avoided upsetting stakeholders by leaving old structures in place for 18 months. (I hope to have much more on that story in the next couple of weeks, thanks to John Santaferraro, HP senior director of marketing, business intelligence.)

Several people in the BI crowd do talk often and thoughtfully about organizational problems. Maureen Clarry, CEO of CONNECT: The Knowledge Network and longtime TDWI instructor, teaches “Power, Politics, and Partnership in Business Intelligence Projects” at every TDWI conference. Participants see for themselves how position shapes behavior. Those short-sighted silo keepers, for example, could flip into data-sharing maniacs if assigned a different position.

Jill Dychè, partner at Baseline Consulting, teaches “BI from Both Sides: Aligning Business and IT,” with strategies to avoid or pave over organizational potholes. She suggests, for example, dodging the perception that BI is “so much data loading and report provisioning.” She writes in email, “We find that the extent to which BI is viewed as a program — with platforms and tools merely components — is the extent to which BI teams are productive and visible in their companies.”

Wayne Eckerson, director of TDWI Research, also addresses these issues, most colorfully with his idea about “purple people.” They are a little bit business-blue and a little bit technology-red, and the purple coloration they acquire lets them traverse the IT-business rivalry.

Wayne spells out some important characteristics for this job, such as maturity and knowledge of technology and business domains. The best are “switch hitters,” by which he probably means to imply that they’re persuasive wherever they stand. In fact, “purple” sounds like a euphemism for another “P” word that Jill actually spells out: politician.

Bad word or not, it’s a critical function. A good politician’s essential function is to coax rivalrous parties into agreement. If that’s the kind of function Wayne sees for the purple people, then they really are, as he says, “the key to BI success” — at least at one level.

Purple may not help much at higher levels. Wayne’s knowledge of of business intelligence is far deeper than mine, but my experience elsewhere makes me think these people are just one of many keys. When I was a sort of purple person myself — in the late ‘90s, bridging an arrogant Web development group and a couple of marketing groups accustomed to full control of their media — my own skill at listening, negotiating, and arm-twisting was only one key. Another key was my boss. At first I had a strong one, later I had an indifferent one, and even later I had virtually no boss at all. I felt like my district shifted boundaries each time, my agenda with it.

One friendly executive suggested I stand up and promote the Web project around the company at any meeting that would let me. He said, “Show ‘em how great it is, and the credit will rub off on you.” Just like a politician running for office.

If I were a purple person today working in BI, where would I go after I’d exhausted training by Maureen, Jill, and Wayne? Most likely, I’d turn for inspiration to books on politics and influence, such as biographies by Robert Caro. Actually, I’ve gone there already, but only because to me politics is a good word. No, you don’t want to emulate Caro’s subjects, just clean and adapt some of the principles they used.

One thing seems clear to me: If purple people, would-be purple people, red people, and blue people are to expand the BI horizon, conversations have to go longer and deeper into the people problems. We start by ending the prissy avoidance of that word that at its best connotes people, perceptions, and compromises: politics!

Does it seem to you like data analysis is busting out all over the place? It might become another fun game like chess or Chutes and Ladders — so this might be good time to recall an old admonition: Don’t just consume data, mind its pedigree.

Repeating the warning, though, makes you look like a party-pooper. In 2007 at the TDWI conference in Las Vegas, a keynote speaker raised it one morning. Jonathan Koomey — author of Turning Numbers into Knowledge and one of those voices the BI world needs more of — did his best. But I could see the unfolding disaster from my banquet table, as attendees glanced at each other in scorn. When the lights went up, not one person raised a hand with any question or comment.

Now Sex, Drugs, and Body Counts: The Politics of Numbers in Global Crime and Conflict, edited by Peter Andreas and Kelly M. Greenhill, tries it again.

You may wonder what sex, body counts, and politics have to do with data analysis, but try to keep an open mind here. The book promises to let us spit out the usual cud of business intelligence, data quality, and get to the real spice: the politics of data. I can’t wait to read it. For now, see Jack Shafer’s review on Slate.

I won’t be surprised if the book points out how each organization’s core group subtly chooses the stories its data tells. I’ve just finished Art Kleiner’s Who Really Matters, which goes into detail on these groups’ formation and influence, including how they define who’s in, who’s out, and why. It’s the essence of politics.

Though core-group members may not ever lay their smooth palms on any data, data is nonetheless coiffed to suit these people. Through layers of managerial interpretation and re-interpretation, their influence cascades all the way down to tiny decisions about how data’s summarized, what’s measured, how it’s measured, and who measures it.

Like other forms of expression within an organization — speech, email, jargon, attire, hair style, suit or T-shirt — data is part of the politics. Though this has a big effect on decision making, it seems rare that I find it on a BI-event agenda. BI’s scope needs to widen.

What can explain business intelligence’s poor adoption rate? Are tools not easy to use? Or is there a deeper reason?

A book from 2000, The Social Life of Information by John Seely Brown and Paul Duguid, suggests that BI designers have neglected basic human needs. Jack Vinson, of Knowledge Jolt with Jack fame, has just posted a worthwhile review that sent me scurrying over to Amazon.

Failure begins early for many new, supposedly revolutionary information systems. Designers “assume that the way people operate with respect to information has to do with only the information. … But there is a social life that revolves around the information that is much harder to capture and codify,” Vinson writes. “We look to verbal and physical queues for validity of what someone is saying. Our business processes have much more than just the inputs and outputs.”

Jumping forward but on the same thread:

… in the essay on reengineering … the authors describe how all the social life around business process is downplayed and often treated as waste. Businesses were re-engineered to remove much of the social lubricant that helped business flow. The essay on knowledge management was hopeful that KM would be a shift away from the intense focus on information and account for the human aspects of knowledge: that knowledge requires a knower. They have a great phrasing: information can easily be written down and transferred. But it is much harder to detach (and transfer) knowledge from the know-er and the context in which that knowledge resides.

The book is still important even after 10 years. It doesn’t even mention business intelligence, yet it addresses some of its fundamental problems.

Take a look at The Social Life of Information on Google Books. I also recommend Knowledge Jolt with Jack. Always worthwhile.