The people who translate raw data into meaning — sometimes titled “data analyst” but often not — are some of the least understood in business and the most valuable. If you’ve read much of Datadoodle, you know that I’m trying to understand them better. Now I’ve launched a survey.

If you play any part in data analysis, please take part in the survey. You may analyze data or you may be a champion of data analysis, or both. You may or may not call yourself a “data analyst.” You may not even do any of this at work, only at home.

For your trouble, you get a preview of the final report when the preview’s ready.

Please go now and take part. Tell your friends, too.

Search all the business literature you can and you’ll never find data analysis compared to romantic love. But, hey, why not? Love’s trajectories might hint at what the business world’s newly enabled generation of data analysts can expect.

These data analysts tend to be independent, are often creative and at least partly self-trained. They’re strapped to rockets from Tableau, Lyzasoft, Predixion, and others, tools that are at first deceptively toy-like. Aren’t they analogous to the garden variety teenager? Bothg groups revel in newly discovered tools, while both pursuits are fundamentally social — as Lyzasoft CEO Scott Davis observes about data analysis. His blog post got me thinking about this.

Everyone shows up ready to rumble. They’re fascinated with the possibilities, they experiment in private, later they have a blush of quick results followed by a long trail of self-training on finer points.

Each group’s toolset is potent and designed for early success but never early mastery. They make lots of mistakes. In love and analysis, people fall for the wrong data, mess up good data and dates, do all kinds of things they wish they hadn’t.

Without realizing, they face danger. I’ve noticed that behind most good trends comes a rotten sibling right behind it. Think of the history of other social events: Hippies begat the Summer of Love and then came Altamont. We celebrated “free love” and then came a surge of sexually transmitted diseases. Baseball begat the World Series and then came batters on steroids. PageMaker begat self-publishing but then came the ugliest lost-cat posters ever tacked on a telephone pole.

You may already wish that bad analysis would go away. Pete Warden, for one, warns of some fabulous ways people trip over new data. We could easily call this stuff “data porn” and ignore it.

But there are even more treacherous pitfalls. These potent tools can change everything in a flash (at the “speed of thought”). One minute you’re in orbit, and the next minute you wish you were dead. With sex comes the hazard of a painful breakup, and with data analysis comes the danger of unwanted speech that’s too hot for any public platform. Oops!

We have ways to deal with all that, but it’s never pleasant. The rejected lover picks up and leaves, and the analyst just finds his creative viz zapped off the cloud — by those who are themselves learning a new role.

The lover and the analyst both feel hurt, perhaps betrayed. Wasn’t each playing by the rules? Wasn’t each part of the group? Suddenly each one feels rejected for reasons that a hasty explanation doesn’t quite calm the hurt feelings.

In hindsight, we realize we shouldn’t have been surprised. Social pursuits can be like this.

By the way, who said good tools were the end of the story? Well, most vendors did. Some teenagers think so, too. But even slightly more advanced users know that technical proficiency is only the price of entry. We do the real work in many long conversations and collaborations with words, data, gestures, misunderstandings and reconciliations, and on and on.

Here the analogy breaks. The tools will keep getting better while the bodies fall apart. But the lesson’s the same: Tools enable, but conversation — better known in the business world as collaboration — is really at the heart of our work.

One of the pleas in Mark Madsen’s fascinating keynote at the TDWI conference in Las Vegas was to let the “Barts” work. The Barts are, of course, the Bart Simpsons among us, the sometimes nerdy rebels who actually come up with interesting analyses and other useful things.

When the lights went up, a “Bart” was right nearby me at the big round table. Though he “loved” Mark’s salute to his work, consultant Rick Paul wanted more.

As a Bart, he works with a lot of “Marges” and “Homers.” In Mark’s model, the Homers are the everyday business intelligence consumers, about 80 percent of most work groups. The Marges are about 18 percent, and they actually think a little. The last 2 percent are the Barts, the ones who analyze and invent — and who’re limited by the BI systems built for Marge and Homer.

The more painful obstacle facing many Barts, says Rick, isn’t about any technology.

He tells how his team started with three people, all data architects, all smart. “We could do anything,” he recalls. Now the team has 120 members, many of them Homers. They’re of the 80 percent who consume but don’t invent or even think very much. “They’ll fake inability,” he says, “to tempt or coerce the three innovators to do their work. They say, ‘Oh, you’re so good at this. It’ll just take you a few minutes to do this.’” It really does take only a few minutes, but “it’s not thinking work.”

“I’m lazy,” he says, “but when I don’t want to do something, I figure out how to automate it.”

Rick says he’s still trying to figure out how this situation can be resolved. He mentions isolation, but he also thinks of encouraging the 80 Percenters to have some vision for their own careers. They should have some way to “add intelligence to their own work on a daily basis. They should be actively engaged with their work.”

Smaller teams might also work, he says. In a team of 120, it’s easy enough to do nothing for weeks at a time. It’s much harder in a team of, say, seven members.

“The innovators have to be positioned to influence the company,” he says, “but not be abused.”

Why do creative data analysts meet resistance? I’ve written about them before: many are young, they’re smart, and their credentials might still be slim. But they’re still a value to their employers, no matter who knows it yet.

At least some of it’s about Boomers vs. the up-and-comers. One corporate communications consultant I talked to, Liz Guthridge at Connect ConsultingGroup, has seen all this in action.

“Boomers are still interested in experts,” she says, “people with loads of experience and loads of credentials.”

Liz tells of one client, a Syracuse University alum, who hired a young guy as CIO. The newly hired young man was “impressive,” Liz recalls. Among other attributes, he had done a double major in college of public relations and information technology. This was his first job out of school.

The client, she says, kept belittling his knowledge — even though he was, Liz says, “really solid.” The problem was in the way the new CIO communicated with the boss.

Liz has advice for younger people with important things to tell their Boomer bosses. The first three points:

1. Think of the exchange from the senior person’s point of view. Since boomers love experts, always provide proof points. And not from Twitter but from boomer-recognized sources like the Harvard Business Review.

2. Network from the bottom up. Find out who the senior person listens to and talk to them. No explicit appeals, just talk.

3. Use the senior’s preferred communication channel. The woman who hired the CIO liked to schedule phone calls, not email or text.

The young CIO, she tells me, is still on the job, though he’s cut back his hours.

Lyzasoft wasn’t among the 38 exhibitors in TDWI’s Las Vegas exhibit hall. Lyzasoft sponsored no part of the lunch, and they hired no stage magician. But their buzz was the loudest I heard over the event’s five days.

Others may have heard different buzz because buzz varies. Business intelligence elites gather every year at TDWI’s big Las Vegas event to teach, and they end up schmoozing, too. Over beer, food, and sometimes playing cards, they compare notes.

Is anyone seeking a consensus? I suppose someone might, but the interesting ones just play with ideas, reflect on what others say, make a joke, and think about it. If there’s any “truth,” it develops during a lot of talk and thought, whether it’s about politics, tofu, the future of passenger rail in America, or business. That goes for any kind of conversation, whether the medium is words or data.

In business, the conversation is somehow forgotten in favor of the data. But to Scott Davis, CEO of Lyzasoft, the conversation is critical to understanding the data. “A chart has no context at all,” he said in mid February. “The conversation is what’s really valuable.”

The conversation-free, top-down “single version of the truth” isn’t always useful for those who need to manage data for specific uses and contexts. Its “truth” may in fact be no better than Soviet planners’ forecasts of market demand for women’s lingerie. “A single version of the truth,” said Third Nature research director Mark Madsen in Las Vegas, “is true for a single beat of the corporate heart.”

Enter Lyza 2, Lyzasoft’s new version of its data-wrangling and collaboration tool made for data analysts determined to create truth for specific uses and context. The first edition of Lyza offered Excel-like personalization. In the new edition, collaboration seems to have been the guide.

You could see this year’s improvements coming in last year’s email from Lyzasoft CEO Scott Davis: “Even though they are quants, their world is personal,” he wrote. “Relationships are vital. They think in terms of ‘who do I know who knows X type of information sources?’” He could have also been talking about journalists, artists, and anyone else who has to hear signals within noise.

In the new edition, Lyza encourages fluid interactions with a variety of social-media tools: email, Twitter-like messaging, SMS messaging, bookmark collections with annotations, and other tools track and fortify discussion. Lyza lets people work easily with other smart people they trust. If “Steve” believes that “Brian’s” work is good and “George’s” work is not, he can work with only Brian’s data. It also publishes to the new tool, Lyza Commons, for even greater collaboration while retaining users’ ability to interact with data. Lyza 2 loves a good conversation.

The data and everything that happens to it gets tracked automatically. Unlike in Excel worksheets, changes are transparent. Automatic documenting allows any change to be dug up and fixed. If only the data-free conversations in politics and other parts of business had such a tool.

I was surprised to hear spontaneous praise for Lyza’s new version. eLearningCurve education director Dave Wells and Third Nature principal and one of the event’s keynote speakers Mark Madsen both did. I heard the same from several other BI experts, too. Madsen even gave a brief look at Lyza in his Executive Summit presentation on the future of BI.

I harmonize with people who appreciate Lyza at least partly because I think it’s smart to let people work the way they want to work — the way people have always worked. They prefer working with people they trust and with tools that respond. Everything else is static.

What’s the hardest part of training a new data analyst? Resetting the trainee’s mindset.

“They start out with the idea that there’s a right answer,” says Joe Mako.

Joe’s leaving his job — where about one year ago he began analyzing data — to go work for the producer of Lyza. Lyzasoft CEO Scott Davis sees him as a “prototype” of a kind of creative, resourceful analyst that Lyza was designed for. Joe will engage with other analysts to evangelize Lyza and to help new users ease into the flow.

Joe, 29 and a veteran of two Army tours in Iraq, started out on the help desk. He answered calls from within the company, an ISP. Many callers couldn’t or wouldn’t analyze their own data, so Joe did it for them. His boss also enlisted his help — and now won’t dare go without a backup.

The first people he’ll help get into the flow are the two women who’re replacing him, and he’s got to do before he starts at Lyzasoft on November 9. They’re some of only a few in the his group who applied. Most others refused the “boring” work with “ugly” data.

New users, he says, want to know, “Where’s my wizard?” There is none. “But that’s why I enjoy these tools.” He uses Lyza and Tableau primarily. “They stay out of my way. They enable me. It’s just me and the data. … That’s what’s neat. But [new users] don’t know where to start.”

“I’m handed crazy files without any structure,” he says. The first thing new users have to know is that, no matter how ugly the data may be, it really can be cleaned up. He demonstrated to his new trainees, he says, and “they were blown away.” After that, he started showing them how they can clean up data on their own.

He explained basic steps and functions. Then he showed them how to combine tools, such as how to use two functions in sequence. And deeper still.

“It takes time playing to figure out where you need to get to,” he says. “You have to just go and play. If one thing doesn’t work, you try something else.”

“I always thought that data was exact,” he says. “If not, it was garbage and I’d throw it out.” But he later learned that there’s usually only a portion that’s garbage — that somewhere within the crazy mess there’s a story. “Even if every data point is wrong, there still might be some trend you can see. If there’s a bunch of ugly data, how do you figure what he story is?” It takes a willingness to figure it out, to untangle it, to find out what’s in there.

That’s a skill, not a talent, he says. “I’ve watched [his two replacements] get it closer and closer, learning to merge other data in, to reshape it and finally produce the output.”

Closer and closer. Business will trudge ahead, training a Joe here and a Joe there until people don’t complain anymore about boring work with ugly data. Someday, many more people will welcome the chance to do this work.

So many of us who feel drawn to visual analysis can’t understand why everyone can’t see the value. “Pretty pictures,” the skeptics mutter. On Eager Eyes, Robert Kosara makes important points that I haven’t seen before.

Toward the end of his post he writes, “We need a new term.” He rejects the aged and indefinite “visualization” and the baggage-laden “visual analytics.” He prefers “visual analysis.”

Whatever we call it, it’s harder to use than it seems.

You have seen the bar and pie charts, but do you actually know what they mean? Do you know how to use them to tease the relevant information out of your data? Can you handle more than two dimensions of data and still find meaningful structures? There is so much more to visual analysis than what Excel offers you.

Good, but then he’s not clear. He writes, “The key problem is that people are much more interested in clicking through interesting pictures than learning about actual analysis work done using visualization.”

Which people? He can’t mean the ones who actually analyze. He must mean the casual users, the data consumers, the armchair analysts — and they will always click through. He writes that those who value visual analysis have to fight the idea that it’s just pretty “or risk the trivialization and marginalization of visualization as an analytic tool.”

You’d think the tide was coming in an threatening a sand castle. But from everything I’ve seen, genuine visual analysis seems to be more and more popular. Even elementary visual analysis works better than the ugly alternatives.

Who are we fighting? The ones who don’t care and never will? No, they’re no more a threat than fast food is a threat to good food. To most people, fast food is good enough — and so are pie charts.

The ones to watch out for are those who sell fast food under the good food banner — the ones who’d propagate sloppy techniques and call it visual analysis. That’ll really spoil our appetite.

For more on “good food,” don’t miss the “Information Visualization Manifesto.”

“Heroic analysts” and journalists keep running into each other, at least in my mind. I realized that two scribbles from last Sunday about journalism can also be about data analysis:

• “… the most precious gems gathered in any journalistic journey are frequently those found around the edges of a story.”

• ” … the secret to memorable nonfiction is so often the writer’s readiness to be surprised.”

They’re both from the same book review. Come to think of it, both belong in the Datadoodle manifesto.

I hear a story like the one I heard this week and I want to ask the apparent villain why. There must be a reasonable explanation.

At first glance, he’s like other managers I’ve known of who throttle promising work for what seems like a personal need for control. “So tell me,” I’d like to say over beers, “what were you thinking when you denied that analyst free access to that data? What’s your side of it?”

If beer didn’t work, I could imagine waterboarding, even though his explanation would then be less reliable.

Here’s the story: One young analyst’s boss wanted an answer to a simple question: How many customers are receiving premium service and paying standard rates? (Sorry, I can’t identify the company.)

The analyst had little training. He was only the most curious, persistent, and resourceful member of his department. He also knew a little SQL. But, as he puts it, “I didn’t know a dimension from a measure.” But despite scant knowledge and many obstacles, in a few months he won the company just over $500,000 in unforeseen revenue the first year.

First stop: the business intelligence system. But problems showed up quickly. While the billing database — the only one within the BI system — listed about 20,000 customers receiving premium service, the installation database listed 25,000, and the customer database listed 30,000. Each should have had the same number.

He knew why they contained bad data. Each new customer triggered a 45-step process that required customer service representatives to enter numbers in correct fields and to choose from drop-down menus. Errors came easily.

The IT manager seemed to shrug at this. He refused the analyst access to the desktop client, which would have allowed him to join the other two databases. All the analyst could do was review each record one at a time. When the analyst asked for help to learn the BI product, help was denied. Nor could he have access to the vendor’s help resources.

“They say it’s their database and they don’t want anyone running queries against it,” he said. “They’re afraid it would impact the performance. But all I want to do is pull a table out. Give me an export! I’m not asking for a complex query. It’s not sensitive data. In fact, anyone in the company can get at it. They have a strange philosophy.”

He admits his next move was sneaky, but justified. He used an SQL injection, a technique for unauthorized entry, to extract a table from the customer database. That database, he said, looked to him like a “10-year-old’s work,” and had no security built in. The administrator caught on when the analyst slipped up and ran an update. The administrator loudly threatened, the analyst said, “to have me listed as a terrorist.”

A loyal insurgent, maybe. Except for the accidental update, he never altered records. He’s only sent batches of corrected records to other departments.

“My boss was able to prove that I had done a great deal of good stuff in cleaning up the database,” he said.

The greatest benefit was identifying 2200 customers who received premium service but only paid for standard service. That list went to the billing department. Every one of those customers began paying more, at an average of about $20 a month.

Even so, approval for the relatively small amounts to buy the two tools he’s found most useful in trial versions, Lyza and Tableau, has taken months.

Insurgents often annoy veterans with their ideas and can-do spirit. I can imagine what the IT manager might say in a candid moment. “The little bastard thought he knew more than we did,” for example. “He would just mess things up.”

Or he might say, “All our data’s screwed up. You think I’m going to let him see all that, tell his boss, and let everybody know?”

He might also say, “I’m a jerk. I’ve always been one.”

Next stop: reasonable and articulate friends of mine who work in IT.

I picked up the term “creative analyst” in late June on the phone with Lyzasoft CEO Scott Davis. But what does he mean?

He described one analyst he’s known of. This guy arrived at a new job with strong recommendations for his ability to tear apart a dataset. He could slice, dice, build related charts and pivot tables — but only with canned data. That is, data someone had given him. This analyst struggled with synthesis — blending separate datasets, for example, or making a formula to derive values, or simply experimenting and asking unforeseen questions.

The ability to improvise and create something new is a “prime differentiator” among analysts, says Davis.

Many of these creative, synthesizing analysts, he says, also tend to feel they have a personal brand. They have a style of charting they prefer, for example, and they produce a distinct set of information that is uniquely attractive to their subscribers within the company.

“You can sort of think of them as publishers,” he says. “They create these things that are in some ways more useful than reports from the BI tool. And they gauge their effectiveness by how many people follow them.”

Such lists have been around quite a while. Before PCs, people did the same kind of thing in hardcopy, producing a dozen or two binders with a distribution list clipped on the cover.

There’s a future, too. Davis expects to see Enterprise 2.0 — social networking within businesses — grow fastest among these analysts. They already have the social habits: commenting, trust, wikis, etc.

He says, “A spreadmart is nothing but a primitive social networking mechanism.”