What the early days of desktop publishing may tell about today’s self-service BI. Read all about it in my latest column at Information Management.

Just by chance, two ostensibly unrelated items arrived here at Datadoodle headquarters on the same day last week.

First, the Department of Here and There received notice of a comment on a post by Information Management editorial director Jim Ericson about his video interview of eLearning Curve director of education Dave Wells. In the interview, Dave described a convergence of forces, a “perfect storm,” that will overwhelm IT shops. Comments ensued, including this one:

… I only hope those who refuse to be intimidated by “IT buzzwords” understand that it’s not going to happen by magic … someone still needs to know what they are doing … and it ain’t glamorous.

Mere hours later came this from the editor on duty at the Foreign Desk. It’s an excerpt of a conversation intercepted in early February between then-ruler of Egypt Hosni Mubarak and an unnamed fellow Middle East dictator presumed to be Muammar Qaddafi:

“I’m tellin’ ya, Mu [Muammar], those kids [the protesters] have no idea what it takes to run a country. Look at ‘em down there [in Tahrir Square]. What a mess! They can’t even pick up after themselves and they think the garbage will still be picked up if I take early retirement!”

I assume the IT person who wrote the blog comment would have nothing to do with the former Egyptian dictator, and vice versa. But the two do have things in common.

First, both have a point. They know that running their vast machinery, whether it’s an IT infrastructure or a medium-size country, is a task for which the barbarians at the door are under-qualified.

Second, some among their peers feel sorry for themselves. Oh, poor us, no one appreciates us! It’s a dirty job, etc. You’ll appreciate us when we’re gone!

But the smarter IT people, like the more sophisticated military professionals of the world, will see this “storm” for what it is: merely a transfer to civilian authority. The smart ones don’t want to run the show. Advanced organizations and countries have managed to shed that spooky awe of power that the less advanced project onto things like the military and technology. This is a step up.

IT budgets may decline and come under more scrutiny. Business people will take more responsibility for some traditional IT tasks, and things may get messy for a while. But isn’t that as it should be?

In any case, the Datadoodle editorial council met overtime on this one. As they say, it’s “a breaking story.”

Wipe away that tear you have shed for BI marketing. Take heart in this: The golden oldies — those tired verses like “faster, better decisions” — have never come closer to receding into the support roles where they belong. A new strategy has been proving itself able to hook even onlookers who swore they really didn’t give a damn.

In the most recent example, a simple Tableau Public-hosted chart on Wired caught me looking. And thinking. It hooked me with a bar chart that compared rates that iPad users pay for downloading data.

Did I say that I really don’t care what iPad users pay per gigabyte?

I looked. There’s the U.S., I thought when I saw it, losing again. I’m used to that by now. But who’s losing worse? Belgium! Why Belgium? I thought of reasons, but none seemed to explain why a gigabyte there was more expensive than a gigabyte in Italy, another country I know a little bit about. Better price supports for waffles than for cannoli? My local Belgian thinks she knows the reason: “Taxes!,” she says. But I told her that that explains nothing at all, and that conversation continues even today.

Sure, it’s all a lot of fun. It’s powerful, too. The simple chart that can hook you on the fly — by making you notice, making you scratch your head, making you try out one angle after another in the quiet of your moment’s pause — can hook customers with modest budgets and legions of casual users to excite.

Back when secretaries were common, you could have had yours track your day in 15-minute increments. In his book The Effective Executive, Peter Drucker suggested this as a way to find out what you really did all day. The picture usually wasn’t so pretty.

Tracking your time then and now is personal, it’s messy, and it’s the essence of business intelligence: collecting data and reading it for guidance in business activities that matter. Is there anything that matters more to an organization than productivity of its people? For a small office or home-based business, this might be the best BI there is.

This gets no recognition in the BI industry that I can find, at least not in the conservative world of TDWI. At least not yet.

PI — for “private intelligence” — has different issues, starting with data collection. In BI, data comes from transactions, all recorded routinely. In PI, most of it has to come from a “secretary” or from our own, tedious notation.

I dabbled in it once. The insights were good, if painful, but mostly it was tedious. A few years ago, a confluence of personal events let me do something I’d always wanted to try: hole up for a few months in a Sicilian village I knew slightly. The food was good, I had relatives nearby, and the nearby church bells rang all day and all night, four times an hour. At the same time, I had a book to edit. To stay productive, I made a game out of the work, tracking my time to the minute in Filemaker.

I liked the local food and started to hate the book, an office manual that inadvertently revealed a con game. Even so, I threw myself at it every day. But no matter how hard I tried, no full day ever resulted in more than about two hours of actual, productive work. My “quick breaks” for walks and coffee with a friend actually took up more time.

I made a Filemaker database because I could find no off-the-shelf product that would do anything close. Each period, no matter how short, had a starting and ending times I entered with buttons, and a calculation field figured the duration. A drop-down menu offered my usual activites. I could make a report for any period.

I thought some product would do that better, but I could find nothing. Then the May 2 issue of the New York Times Magazine ran an article by Gary Wolf about this, “The Data Driven Life.” My Filemaker invention wasn’t too far from what others have used, and now new devices are coming along that could make all that seem so old hat. Some people are even sharing their data on the cloud.

But as in traditional BI, the technology just gets you in the door. The show has just begun.

Most people Wolf writes about do it for personal reasons. One wanted to know how his coffee consumption helped him focus, another tried to cure his sleep apnea, and still another noticed that flax seed oil, or just lots of butter, improved his cognitive performance.

As in good BI, the experiments often raised new questions. And sometimes the new questions are unexpected, as in Wolf’s own experience.

Often, pioneering trackers struggle with feelings of being both aided and tormented by the very systems they have built. I know what this is like. I used to track my work hours, and it was a miserable process. With my spreadsheet, I inadvertently transformed myself into the mean-spirited, small-minded boss I imagined I was escaping through self- employment. Taking advantage of the explosion of self-tracking services available on the Web, I started analyzing my workday at a finer level. Every time I moved to a new activity — picked up the phone, opened a Web browser, answered e-mail — I made a couple of clicks with my mouse, which recorded the change. After a few weeks I looked at the data and marveled. My day was a patchwork of distraction, interspersed with valuable, but too rare, periods of focus. In total, the amount of uninterrupted close attention I was able to muster in a given workday was less than three hours. After I got over the humiliation, I came to see how valuable this knowledge was. The efficiency lesson was that I could gain significant benefit by extending my day at my desk by only a few minutes, as long as these minutes were well spent. But a greater lesson was that by tracking hours at my desk I was making an unnecessary concession to a worthless stereotype. Does anybody really believe that long hours at a desk are a vocational ideal? I got nothing from my tracking system until I used it as a source of critical perspective, not on my performance but on my assumptions about what was important to track.

I wish Drucker were around to respond. Wolf’s insight sounds like important stuff for everyday knowledge workers, especially those who work alone. What’s more important to a knowledge worker than time?

These experiments are often haphazard and highly personal.

Generally, when we try to change, we simply thrash about: we improvise, guess, forget our results or change the conditions without even noticing the results. Errors are possible in self-tracking and self-experiment, of course. It is easy to mistake a transient effect for a permanent one, or miss some hidden factor that is influencing your data and confounding your conclusions. But once you start gathering data, recording the dates, toggling the conditions back and forth while keeping careful records of the outcome, you gain a tremendous advantage over the normal human practice of making no valid effort whatsoever.

Yes, just as analytics gives companies a tremendous advantage over those who make less effort.

Let the BI traditionalists pooh-pooh self-tracking. The very same people might have dismissed such things as visual analysis, agile development, and at one time even business intelligence itself. Sometimes it take a few pioneers and geeks, perhaps even a secretary, to prove a concept.

Now comes the hard part in business intelligence: figuring out how the humans can make better use of all our data and tools for decision making, writes Wayne Eckerson, director of TDWI Research. Let’s bring in the shrinks.

When Wayne points to a trend, it’s news even if others might have already foreseen it. He’s one of the industry’s most thoughtful observers, and one of the most deliberate.

In Tuesday morning’s blog post, he suggests improving BI by enlisting those who study how people make decisions.

To take BI to the next level, we need better insights into human behavior and perception. In other words, it’s time to recruit psychologists onto our BI teams.

He gave an example of one place that could have benefited from visits to the shrink’s couch.

A recent article in the Boston Globe called “Think Different, CIA” provides some instructive lessons for companies using BI tools to make decisions. The article describes a phenomenon that psychologists call “premature cognitive closure” to explain how humans in general, and intelligence analysts in particular, can get trapped by false assumptions, which can lead to massive intelligence failures. It turns out that humans over the course of eons have become great at filtering lots of data quickly to make sense of a situation. Unfortunately, those filters often blind us to additional evidence — or its absence — that would disprove our initial judgment or “theory.” In other words, humans rush to judgment and are blinded by biases. Of course, we all know this, but rarely do organizations implement policies and procedures to safeguard against such behaviors and prevent people from making poor decisions.

See his full post here.

Be sure to see the comments, too. He writes in reply to questions, “Like data governance, we need some principles for approaching and managing decisions. Maybe we should start a decision governance institute!?”

I can’t help notice: an institute.

See “CIA’s insights on the psychology of analysis” on Datadoodle.

With all the talk about technology, let’s pause to refresh with three basics — culture, conversation, and collaboration. These basics will take center stage in BI’s future, and they’ll help decide which tools dominate.

Lyzasoft CEO Scott Davis and I have been reading the same book, The Culture of Cities by Lewis Mumford. The great thing about Mumford, said Scott, is his mastery of so many subjects, which lets him see relationships and meaning among facts that might otherwise seem irrelevant to each other.

The intelligence that soaks up through every paragraph makes the book a thoroughly enjoyable and insightful chew.

“Think of the most intelligent people you know,” he said. “Are they intelligent because they have an encyclopedic knowledge of facts? Or is it their ability to see relationships? I would say it’s door number two, right?”

Trouble is, we can’t all be Stephen Hawking or Lewis Mumford. That calls for linking of minds. That is, collaboration and conversation. We have to find a way for normal people to find a way to see relationships and offer them.

Train people?, I suggested. Maybe, he replied, but what would that say about the state of BI? It’s natural to have conversations and collaborate. Relevant knowledge and observations bubble up as people focus on something and make associations, gradually raising the group’s insight.

But today, BI platforms make contributing difficult. “BI designers decide what reports are going to be out there, and that’s the well you can drink from.” Instead, people should be as free as in any conversation to make new syntheses, to comment, to recommend — “all the things you’d expect at a dinner party. BI doesn’t feel like a dinner party, does it?”

“I suspect,” he said, “that even though we don’t see it within formal tools, it really is happening, such as through email, spreadmarts, the water cooler and such.”

First comes culture. “If you don’t have that culture that draws people into that practice [of collaboration],” said Scott, “you can have all the tools in the world and it isn’t going to help.”

Next come tools. “There are technological things we can do to make contributing more routine.” He mentioned two mashup tools he likes: JackBe and NetVibes.

The entrenched players are going to change, he predicts. Either Cognos and BO and the others will become a lot more like JackBe and NetVibes, or else companies like JackBe and NetVibes will become the dominant players.

He said, “Ultimately, open and relational are going to win.”

The shouts from the back of the BI room seem to be getting louder. In various ways, they’re saying let Big BI die.

Former TDWI education director Dave Wells, visual analytics critic Stephen Few, and Tableau Software CEO Christian Chabot are back there. Others, too.

Last spring, Wells proposed a new, people-centric definition of business intelligence. (See my Q&A with him for TDWI.). On November 18, he published “The Changing Face of Business Intelligence.” He predicted that the industry will soon “experience change that will have broad, deep and lasting impact.”

In early January, Tableau CEO Christian Chabot talked about the failure of BI platforms to deliver the BI promise. He seemed to laugh in surprise at interviewer Carl Weinschenk’s comparison of Chabot’s thoughts to Ronald Reagan’s 1987 challenge, “Mr. Gorbachev, tear down this wall.” The comparison was apt.

Now last Thursday, Few sounded like he’d had about enough with BI tools “entrenched in a techno-centric paradigm.” He wrote in his weblog, “It’s time for traditional BI vendors to pass the baton.”

I believe it’s now time for the vendors with real decision support solutions to thank the BI industry for the technical infrastructure that it’s provided, but then set themselves apart as a new industry, different from but complementary to BI. Much as groups of people throughout history have arisen and set themselves apart to fix what cannot be fixed within the reigning power structure, the decision-support solutions that people need will only make their mark on the world by leaving the calcified fortress of BI.

Thomas Jefferson might have put it this way.

When in the course of business it becomes necessary to dissolve the bands which have connected us to failed technology, after a long train of abuses and usurpations, it is our right, it is our duty, to throw off such technology and to adopt new tools for decision making.

Big changes can seem to come suddenly. The Berlin Wall, the Soviet Union, the Tech Bubble of the ’90s, and Lehman Brothers all came down to widespread surprise.

I don’t put much faith in predictions. But these three independent thought leaders are all pointing in about the same direction. I’ll be looking for changes coming that way.

We forget most failed predictions quickly. If you make a bad one, you just say “let’s move on,” and you’re as good as moved on. But sometimes you meet the kind of guy I say hello to near my office—the kind professional forecasters hope they never meet.

When he and I get stuck in line together at the little grocery store, we talk mostly about the weather, and lately the forecasts for rain have been wrong. The other day he grumbled, “And they never apologize.”

The weatherpeople never apologize? I’d never heard that before. What if there are more of these guys out there—reading my trend predictions for 2009?

Last January, after of having to predict trends for 2008, I bought a book about forecasting—actually, fortune telling, which is similar. The Full Facts Book of Cold Reading (2005; Ian Rowland) is practically a how-to for would-be psychics. I read about it in a New Yorker article’s unflattering comparison of the FBI’s profiling operation with storefront psychics.

The trick, essentially, is in the phrasing. Say it so you’re right no matter what the truth is. It works because most consumers want to believe you. But you knew that.

In early 2009, when all the forecasts are all in, when the last drop has fallen, I’m going to put them all together to look at the trends foretold by all the trend-spotters. All in one bucket. Bucket of what? We’ll see.

It’s hard to see through the smoke as our financial house burns down, I know. But what I’ve noticed is more interesting: the first signs of rebuilding.

This month, three experts I read—visual analytics expert Stephen Few, Competing on Analytics author Tom Davenport and digital-media economy specialist Umair Haque—all seem to have knit recent blog posts with the same thread: honest value in business and the economy.

“The macro crisis isn’t really a financial crisis, an economic crisis, or a solvency crisis,” writes Haque. “It’s an institutional crisis: the economic institutions of capitalism are in shock.”

Perhaps it’s the shock of having been conned. One of the three levels of transformation he suggests is the return of authentic value.

Authentic, long-run value isn’t created through arbitrage or gamesmanship — what we too often confuse strategy for. Games of off-balance sheet accounting, currency hedging, capital structuring, so-called labor arbitrage — where corporations simply shift to the lowest-cost, or most poorly regulated, sources of manpower — don’t create value. They just shift it around. Corporations who play this game of economic musical chairs are in for a rude awakening – because the music just stopped. And so they must rediscover the simple fact that value creation flows from making economic activities not just profitable in the short- run — but meaningful over the long-run.

Davenport lets some air out of “fluffy” social networking. He asks, “How can we really be producing value if we’re all sitting around blogging and Facebook-friending each other?”

All this fluffiness will be hard to maintain in our next period. He writes that “it wouldn’t be a bad outcome if the current crisis led to a more diligent, industrious economic climate.”

Few takes us down to products. He argues for honest value in business intelligence software. He often confronts badly designed dashboards—adorned with eye candy and other silliness—and urges those who listen to educate customers, not pander to them. The response is too often that, yes, we know, but it sells so there’s nothing we can do. Ah, promiscuity as a business strategy!

Any business that measures its success by current sales revenues or profits without regard for the effectiveness of their products will go for the silly stuff every time. I could argue that this is a poor business model because it’s short-sighted and doomed to fail, eventually resulting in declining revenues, but what’s the point? Businesses built on this model lack the foresight to appreciate the greater intelligence of long-term planning around products and services that effectively address the real needs of people. I believe the root problem that belies such business practices is not strategic short-sightedness or a myopic focus on sales—these are symptoms of a deeper, more fundamental problem. I believe that it’s wrong to build a business on self-interest alone.

Enough with this promiscuity. He adds, “Things that don’t work should not be sold—period. That’s good business.”

If the next president and Congress take Davenport’s advice and nurture big importers, high tech would surely be one of the first picks. How devastating, though, if Indian and Chinese entrepreneurs took advantage of an easy opening and overtook the U.S. even there.

I happened to read Few, Haque and Davenport. But I’ll bet they’re among many others these days arguing for a return to honest value and good business.

Companies are looking to BI tools for help with rising energy costs, says Dan Esty, co-author of Green into Gold and the Hillhouse Professor of Environmental Law and Policy at Yale University.

In this slumping economy, he said yesterday, he’s seen companies defer advertising campaigns and factories in favor of investment in BI.

“They’ve concluded that there is a very substantial return on energy efficiency,” he said, “and the way you find that is with BI tools.”

Energy savings in companies he’s worked with, he said, are no less than around 10 to 15 percent, and have reached 60 percent. IT departments usually yield some of the most improvement.

Collecting the data is for most companies a problem. Will Sarni, CEO of Domani, says that even as companies have done a good job of rolling up their financial data, they’ve done a poor job of collecting energy data.

“You’d be surprised to see how archaic it is for some companies,” he said yesterday. “In some ways we’re still moving out of the Stone Age.”

There’s a big scramble now for tools that can be used easily within an organization to collect and understand and to make rational decisions—and that are relatively painless to use.

What’s missing most, he says, is a way to monitor company-wide energy consumption in real time.

Energy expense is becoming important in smaller and smaller companies. It used to be that an annual expense of less than $1 million was considered unimportant. “Now it’s significant.”

It’s a story in progress for BI This Week.