There are the dream organizations that deploy data analysts wisely. Then there are the nightmares, such as the I.R.S. as portrayed in David Foster Wallace’s last novel, The Pale King, reviewed yesterday in the New York Times.

… In a universe of veiled and veiling numbers, the task of drawing the true [data] out into the light and holding them up for inspection, clear and remainder-­less, really is a sacred one. … The problem, as I.R.S. recruits soon discover, is that neither moral nor heroic codes hold true anymore.

These recruits work with “excruciating difficulty … in an age of data saturation.”

The [instructor] presents “the world and reality as already essentially penetrated and formed, the real world’s constituent info generated . . . now a meaningful choice lay in herding, corralling and organizing that torrential flow of info.”

One character is the data psychic, Sylvanshine, who can “glean trivia about anyone simply by looking at him.” But, as if to prove that good data is far from the end of the story, he has a problem.

[He] is “weak or defective in the area of will.” Nor, due to endless digressions, can he complete anything. No one can; in “The Pale King,” nothing ever fully happens. That this is to a large extent a metaphor … becomes glaringly obvious when we hear one unnamed character describe the play he’s writing, in which a character sits at a desk, doing nothing; after the audience has left, he will do something — what that “something” is, though, the play’s author hasn’t worked out yet.

Let’s see, will an “easy to use,” “speed of thought” tool help? Is there a tool for Sylvanshine and the others?

No, at least not until the next update. But this is why business intelligence is fascinating. Under cover of tools and data, we touch the heart — throbbing or dead — of the organization.

From an email to me this week. Quoted with permission.

I’ve been doing this ‘data analyst’ thing for almost a decade now and I’m not even sure I’m doing it right. Does anyone really know what a data analyst is? I’m a Microsoft Access guy that got lucky and learned SQL from a bunch of very sharp devs. Then I got lucky again when I met [name withheld] and discovered [name of tool withheld]. Now I get to play with data in a creative way. I guess that makes me a data analyst. I know there are better da’s out there that write ugly SQL, but they get the job done so I guess it’s not about coding. I think we’re riding a wave right now. Old bi, New bi … it’s all blurring together. … I don’t know if I’m doing it right. I know that sometimes I help people and sometimes the work I do makes it easier for them to do the work they do.  For me, that’s enough.

A guy walks into your cube and asks you to whip up an econometric model. You’re a statistician, after all, and you’ve got a Ph.D. in something or other. You do this for lunch, he figures.

He “over-thought,” says the one whose cube such a guy walked into. Theresa Doyon has been routinely navigating datasets in the 50 to 300 million-unit range for 10 years. She’s good at all-terrain tools like SAS and KXEN. She could have produced the report he wanted. Instead, she asked him, “Why?”

They talked for an hour. What he actually needed, she discovered, was more of a descriptive report — something that gives a picture of a current situation. All he wanted to do was figure out how to allocate resources.

This is where things usually go wrong. “We’ve got all these great tools,” she says, “but I don’t see us using them as well as we could.” We could write this off as a communication problem, but she believes it’s much more than that.

It’s something like a man who walks into a kitchen with a carton of eggs and tells the cook to fry them all — when actually he’s just hungry and happened to have found eggs. Something else might suit him better. Bacon?

Business people complain that they get reports, not insights. But they’re not sure how to ask for the data. Meanwhile, analysts complain that they’re asked for a fire hose of analysis and that it’s always due yesterday. But they deliver. Asked for a report, they produce a report. Asked for a model, they produce a model.

“But that’s not what the business people really want,” says Theresa. ‘What they really want is to answer some sort of business question, like how’s my marketing doing? What can I do differently?”

“If you had a strategy and you did bite-size tests and learned as you go,” she says, “you could start to use the analytics that would really drive the insight.” We can really improve the way things are done. That’s the missing link.

At least some organizations have found that link. A home furnishings retailer she worked with recently had her work closely with marketing people as their questions and her analysis evolved.

The retailer had been suffering as ever more of its customers feared they’d soon lose their homes. The living room sets that looked so good just months before seemed to lose their appeal. Theresa’s assignment was to, in effect, come up with a silver lining.

“It was a very big and ambiguous question,” she recalls. She worked with the client on a series of projects over nearly two years as the sour economy evolved. She estimated opportunities, customer targets, and gave the marketing people she worked with critical guidance on the launch of new programs. She recalls, “It came out quite well.”

See her LinkedIn page here.

As Stephen Few delivered his keynote address at the recent Tableau customer conference in Seattle, he suddenly broke his rhythm to look at someone in the audience. “Is that Howard Dresner?” he wondered, surprised.

It was. Howard is the man who as a Gartner analyst in 1989 revived the term “business intelligence,” and he’s one of the industry’s patriarchs. He holds a seat on the TDWI faculty, and he founded Gartner’s business intelligence event.

In that world, Tableau is still an insurgent. Tableau usually bypasses IT buyers on its way to data analysts, who only want to soak insight from data and then show others the results. Many Tableau users are veterans of miserable, lockstep interfaces procured by those IT buyers and made by IT-facing vendors. That at least partly explains why Stephen Few’s evisceration of the Business Objects interface seemed to delight just about everyone in the audience. Even Howard called it “wildly entertaining.”

Two mornings before, Tableau CEO Christian Chabot had started the fire. In his Monday morning keynote, he made allusions to Apple’s “1984″ ad, then went into Tableau 6 — using FAA data on another variety of Big Brother: the airlines on which most of the crowd had arrived in Seattle. Both keynotes were as much fun to watch as the wild flames of a bonfire or a public execution.

Now and then, though, I had to check the exits. My inner skeptic always questions such fervor. I subscribe to no guru, no preacher, no righteous political philosophy, nor any movement that’s “green,” “red,” or “blue.” When you glimpse the underside, they’re all too ugly to bear.

But, again and again, I’ve seen that this is not any of that. The Tableau crowd is simply a bunch of people who’ve found a good, honest tool that responds the way good tools do. They have fun listening to the keynotes, but most Tableau users themselves are about as fired up as chess enthusiasts or weekend car mechanics. They’ve adopted something that’s logical, responsive, and economical, in which the simple interface encourages experimentation and learning.

One of the only signs of celebration was giddy tweeting. I suppose that was hard to take for some. One BI biggie — perhaps speaking for the Ministry of Truth — grumbled in a tweet from far away that there was “no silver bullet,” but most Tableau users don’t follow him and don’t care. Others, such as Howard Dresner and Microsoft lead for BI strategy Bruno Aziza both showed up to see what it was all about.

The conference was sold out. Total paid attendance was around 700 — more than twice last year, which was significantly higher than the first year. Most user conferences, in fact, declined this year and last. This year’s total was also in the same range as TDWI World Conference in San Diego, held just two weeks earlier, and not too far away from the 1000 or so that Howard Dresner says Gartner often attracts.

At this rate, says CEO Christian Chabot, the conference will be forced out of Seattle next year and possibly longer. The only space that will hold a larger crowd than this year’s would be space mashups within an easy walk of each other, and that’s not available next year. Tableau is already looking at San Francisco and other cities, even Las Vegas.

One user, now back home in Lithuania, pondered the future: Giedre Aleknonyte, a data analyst at a phone carrier, said “You know how people say they’ll ‘Google’ to find out some information, even when they don’t actually use Google? Maybe someday when we want to analyze data we’ll say, ‘I’ll just tableau it.’”

It was International Workers’ Day on Saturday and the official release day of Lyzasoft’s latest product: its foray into “free.” It’s a good way to say “power to the people.”

Some people associate that slogan with protests and even violence. But I think the best paths to power usually involve well-analyzed data, whether in public life or at work. Now the Little Guy has a potent new tool to deploy.

Lyzasoft founder Scott Davis calls Lyza on Lyza Commons “the YouTube of data.” This fully functional cloud-based version of Lyza is a strong tool for office-based, home-based, cubbyhole-based, dorm-based, or public library wifi-based users and groups. Import your data from whatever sources you have, refine it, share it with to whomever you like, and even charge toll over Paypal if you want to.

“Obviously,” Scott says, “what we’re doing is saying, ‘This thing can scale.’ But instead of going for the uber-enterprise as our leading play, we’re saying that what’s unique about this technology is it can make it to everybody within a small and medium business without having to have a big IT team around.”

Lyzasoft’s second, paid tier serves customers who need private clouds. That version starts at “small” for $150 a month, seating up to 10 users and providing “plenty” of storage. Go upward through “medium” and into “large,” and you pay $2500 a month for up to 200 users.

Wait, you say. You’ve heard this “YouTube of data” thing before. Yes, just three months ago another YouTube of data launched, Tableau Public. (I wrote about it here.) Tableau, Lyza, and YouTube itself all say “power to the people” by popularizing a medium with free, easy-to-use tools and a venue. Each one’s growing crowd of Little Guys and their audiences turns into a movement that the those in executive suites can’t help but notice. At some point, YouTube and those who follow its model hope that “free” leads enough customers to “ka-ching” to yield a profit.

YouTube seems to be well on the way. Its ready-to-roll movie theater had fired the imaginations in a waiting mob. These filmmakers-to-be had been trained over years of TV and movies to understand film and crave a chance to do their own.

Is there a waiting mob of would-be data analysts? One pioneer of free analytics is skeptical. LucidEra founder Ken Rudin, now vice president of analytics at Zynga, says you need more than free tools, no matter how easy the tools are to use. He says, “Tools are only as valuable as the questions you ask.” One of his biggest hurdles was getting customers to appreciate the possibilities of analytics.

But the YouTube idea is more than tools. It’s a game and a self-reinforcing mob. The tiny films YouTube users make don’t just play as if on a jukebox, they’re scored, they’re answered, and commented on. It’s like the difference between voting in a little booth and going out on a street march. It reinforces and stimulates. Unlike most business environments, it asks people to play, which is how Lyza Commons and Tableau Public users will break out into creative and incisive data analysis.

I also hope there’s a new supply of analysts. Ken Rudin and others are hungry for them. (In fact, if you’re a data analyst and you want to work with cutting-edge technology and data in one of the world’s largest databases, email Ken today at krudin@zynga.com.)

Power to the data analysts!

I’m sorry to tell you serious types out there, but visual analysis is often a game — in fact, one of the best games in town with Tableau Software’s visual analysis tool. Now Tableau Public is going to bring it to the masses.

In the same way that YouTube spawned a surge of new filmmakers, Tableau Public — free, running the same engine as its desktop sibling, and embedable — will bring on a new generation of data players and spectators.

I was a spectator at a data visualization conference one afternoon two years ago. Tableau Software director of visual analysis Jock Mackinlay had finished his presentation and another person had started his. Yet someone at the control board forgot to flip a switch, and Jock’s live screen remained on one of the room’s big screens. Jock assumed his screen had been hidden, and he kept playing with the data. I don’t have to tell you who seemed to have the audience’s attention until someone pointed out the problem.

The mere visual distraction was minor. Even without narration, I got caught up in the apparent drama as he tried one look at the data after another.

Not long after that, I wondered aloud to someone at Tableau about data hobbyists. I imagined people who foraged for data to analyze then publicize it to start conversations, collaboration, or duels. Data would be their raw material of choice just as scrap metal is to some sculptors or overheard conversations is to some fiction writers.

There was no such community visible then. But I realized this week that I know one now: Dan Murray, a skilled, dedicated Tableau user. He jokes that he’s a “freak” because he analyzes data from the federal budget and posts his often provocative analyses. He’s already been answered by at least one who disagrees with him.

In beta and since its February 11 launch, Tableau Public has hosted a flurry of visualizations, including these: a map of top venture capital firms investments by U.S. region; a chart showing how long it takes to build a technology empire; a history of earthquakes in Haiti; a neighborhood breakdown of housing supply in Seattle; trends in U.S. high school graduation; and studies of deprivation and marginalization in education. In most cases, spectators can become players by selecting subsets of the data to find answers to their own questions.

With popularity comes some misuse. Many of the charts will break rules, such as what happens in another kind of game, YouTube. A New York film editor I know complains that many YouTube-acculturated film editors have neglected basic editing principles. She writes that they rely so much on special effects that they “can’t put two shots together and have them work as an unembellished edit.” On Tableau Public, there will be pie charts, chart junk, and even baselines that do not start at zero. We’ll survive it.

But what’s all this got to do with the very serious practice of business intelligence?

Like monks must have done when printing presses began producing books for the masses, many priests of business intelligence will stand aside, arms folded in the aspe chapel. But I predict that before long even they will appreciate a wider, deeper pool of analytical talent ripening for training and employment.

I suspect that the new bunch will have been sharpened by the give and take of public exposition. They’ll also learn from playing in a huge community the way artists and craftspeople of all kinds improve their skills when they bump into peers every day.

This is a new clue for the future of BI. It can’t help but improve data analysis in business. So let the games begin.

Dear Datadoodle: My title is “strategist analyst,” but I’ve become just “the data geek.” As soon as I get into the fine points of my data, they roll their eyes. In meetings, they make little jokes to each other, or they just stare out the window. Please help. I’ve got loads of great data but managers have no time for me anymore. The Data Geek

The Data Geek has lots of company, says Christine Muser, longtime data analyst, founder of CyCom Solutions, and writer of CyCom’s weblog Pharma-BI. She’s seen data analysts stumble over this problem. She had to deal with it herself.

“I love data,” she says, “so in my younger days I used to just barrel right ahead into the details.” Too often, though, she’d see only glazed looks in her audience.

It was even worse for one man she once worked with. He actually became useless as a strategist. He spent so much time pulling together data, manipulating it in Excel, and poring over results that he kept managers waiting for results. After a while, those who relied on him got tired of waiting — and tired of his overly detailed explanations about dimensions, data sources, and methods.

Now, Christine analyzes more than the data. “It pays to know your audience. If you know they’re very familiar with the underlying data, you can give more details,” she says. If they’re not familiar with the data, “You can say, ‘here’s what we’ve observed’ and ‘here’s the impact,’ then be quiet and see if you get puzzled looks.”

The essential lesson she learned: “Understanding what is meaningful is a really big deal.” A strategist understands what’s significant, and doesn’t bother with the small stuff.

For example, say you’re analyzing a pharmaceuticals market. Your company makes a drug that treats only one symptom, while some competing companies make drugs that treat that symptom plus several others. That difference makes a straight comparison — pill for pill or dollar for dollar — useless. So you have to adjust the data to allow for that difference. But managers usually don’t want to hear how you did it, only that you did take care of it.

A strategist also knows when to ignore buzzwords. A manager who liked to stay current once asked Christine, “Can you do neural networks?” Perhaps, but dare go into what that would take and you risk running out your clock.

The week before last, I told about the young data analyst who got the door slammed in his data-seeking face, and I asked “why?” This week, a veteran of the data business answered.

He has spent decades at a large company that produces data solutions of all kinds. I’ve known him for two years, mostly as a voice for the technical side of the house.

The obvious answer is that the IT guy in that story is afraid of giving away the jewels. But that can’t be all there is to it, and my IT whisperer explained.

“IT only lives as long as the business side perceives that they have more knowledge,” he said from Texas this weekend. “If the business side thought they could run the systems without IT, [IT] would be out of there and replaced by mamoo [one's favorite pet]. What dollar did IT ever put in the business’s pocket?”

Not even data quality is safe for IT. My source knows a president of a small software-and-services company who told him this summer, “I’m the only one who really cares about data quality. I’m the only one who needs to have all the data hang together. Everyone else is in their own little cubbyhole.” His job is to email others when the data doesn’t all sync together.

We’ll always need data mechanics, of course, just as car owners need people who align wheels and rebuild engines. The difference between now and 100 years ago is that mechanics are now part of the team. So it’ll be in BI.