The people who translate raw data into meaning — sometimes titled “data analyst” but often not — are some of the least understood in business and the most valuable. If you’ve read much of Datadoodle, you know that I’m trying to understand them better. Now I’ve launched a survey.

If you play any part in data analysis, please take part in the survey. You may analyze data or you may be a champion of data analysis, or both. You may or may not call yourself a “data analyst.” You may not even do any of this at work, only at home.

For your trouble, you get a preview of the final report when the preview’s ready.

Please go now and take part. Tell your friends, too.

Search all the business literature you can and you’ll never find data analysis compared to romantic love. But, hey, why not? Love’s trajectories might hint at what the business world’s newly enabled generation of data analysts can expect.

These data analysts tend to be independent, are often creative and at least partly self-trained. They’re strapped to rockets from Tableau, Lyzasoft, Predixion, and others, tools that are at first deceptively toy-like. Aren’t they analogous to the garden variety teenager? Bothg groups revel in newly discovered tools, while both pursuits are fundamentally social — as Lyzasoft CEO Scott Davis observes about data analysis. His blog post got me thinking about this.

Everyone shows up ready to rumble. They’re fascinated with the possibilities, they experiment in private, later they have a blush of quick results followed by a long trail of self-training on finer points.

Each group’s toolset is potent and designed for early success but never early mastery. They make lots of mistakes. In love and analysis, people fall for the wrong data, mess up good data and dates, do all kinds of things they wish they hadn’t.

Without realizing, they face danger. I’ve noticed that behind most good trends comes a rotten sibling right behind it. Think of the history of other social events: Hippies begat the Summer of Love and then came Altamont. We celebrated “free love” and then came a surge of sexually transmitted diseases. Baseball begat the World Series and then came batters on steroids. PageMaker begat self-publishing but then came the ugliest lost-cat posters ever tacked on a telephone pole.

You may already wish that bad analysis would go away. Pete Warden, for one, warns of some fabulous ways people trip over new data. We could easily call this stuff “data porn” and ignore it.

But there are even more treacherous pitfalls. These potent tools can change everything in a flash (at the “speed of thought”). One minute you’re in orbit, and the next minute you wish you were dead. With sex comes the hazard of a painful breakup, and with data analysis comes the danger of unwanted speech that’s too hot for any public platform. Oops!

We have ways to deal with all that, but it’s never pleasant. The rejected lover picks up and leaves, and the analyst just finds his creative viz zapped off the cloud — by those who are themselves learning a new role.

The lover and the analyst both feel hurt, perhaps betrayed. Wasn’t each playing by the rules? Wasn’t each part of the group? Suddenly each one feels rejected for reasons that a hasty explanation doesn’t quite calm the hurt feelings.

In hindsight, we realize we shouldn’t have been surprised. Social pursuits can be like this.

By the way, who said good tools were the end of the story? Well, most vendors did. Some teenagers think so, too. But even slightly more advanced users know that technical proficiency is only the price of entry. We do the real work in many long conversations and collaborations with words, data, gestures, misunderstandings and reconciliations, and on and on.

Here the analogy breaks. The tools will keep getting better while the bodies fall apart. But the lesson’s the same: Tools enable, but conversation — better known in the business world as collaboration — is really at the heart of our work.

How do you approach unfamiliar data? An investment banker I talked to last week — one I know from a client’s whitepaper — rejects the “don’t think” method, advocated in my earlier post about Dan Murray. Instead, he thinks first, on paper.

“My approach is driven by having a bunch of gray hair,” says Michael Princi, managing director of ThoughtStorm Strategic Capital, a boutique investment bank and advisory firm in northern New Jersey. “I want to use my business acumen to tease out what might be the underlying issues.”

Experience counts The naive mind is prone to bad mistakes, he says. Take, for example, the 22-year-old analyst in India he employed who spotted this correlation: a firm’s revenue correlated with the number of Bobs in the workforce.

Hypothesize on paper “I first think through my hypothesis on paper,” he says. “‘It gives you a starting point.” If the model is wrong, as it often is, he just tries another one.

Test and repeat If the actual numbers are somewhat close to his expectations, he knows he’s on the right track. It’s the traditional consulting confirm-or-deny method. When the data does confirm his hypothesis, he’s able to run through the data again and again in iterations.

Test and repeat If the actual numbers are somewhat close to his expectations, he knows he’s on the right track. “It’s the traditional consulting confirm-or-deny method. It’s the quickest way I know. When the data does confirm his hypothesis, he’s able to run through the data again and again in iterations.

Does Michael Princi really analyze data differently from Dan Murray? They’ve never met, but Michael guesses not. He says of Dan, “I think he’s probably mapping it out intuitively.”

Do you have a routine for analyzing unfamiliar data? Please introduce yourself here.

So many people who should know better seem to miss the point when they mention Tableau. Why? I asked BI veteran Stephen McDaniel for his thoughts — which he gave, but then went on to suggest an almost unheard of challenge: a data analysis face-off among vendors.

Consider this description by a BI analyst: “Tableau provides business analysts speed of thought visual analysis on data held in memory on their desktop machines.” All that’s fine, but it may as well have been about a whole bunch of other tools, too.

At the root of this fuzz, explained McDaniel, is that most analysts who concern themselves with tools don’t actually use the tools. They rely on demos , marketing, and hearsay.

Though much of McDaniel’s recent work has centered on Tableau — his second book is Rapid Graphs with Tableau Software and he gives training sessions around the country — he also has a long, credible trail back through BI and data mining. He was director of analytics at Netflix, and has worked with more than 50 companies in BI. His first book was SAS for Dummies.

“I love SAS,” he says. Still, he remembers his sister in law’s reaction to his book on SAS. She was not an analyst but a “people manager.” These are the ones, he says, who have hated BI because “it had been made into a priesthood.” When she had looked through the book, she said, “Oh, this is great” and put it down. But she read the Tableau book for a half hour and said, “You should come talk to some people I work with.” She had recognized what she could do with the tool.

McDaniel’s sister in law and many like her don’t care whether the data is “in memory,” they don’t see themselves as business analysts, they take “desktop” for granted, and they know “speed of thought” is just gloss.

The list of features really doesn’t matter. All that really matters is whether someone can do what needs to be done with the tool.

McDaniel imagines a throw down, a data analysis match. It would be open to any BI vendor. Each vendor would send their best people, and each team would receive a uniform set of data. Over some defined period, teams would analyze and then present the results to a panel of vendor-neutral judges.

The reward? Perhaps a signed copy of a Stephen McDaniel book, or maybe a beer, possibly both. But certainly, repute.

What do you think of the face-off idea? Please write a comment.

Back when secretaries were common, you could have had yours track your day in 15-minute increments. In his book The Effective Executive, Peter Drucker suggested this as a way to find out what you really did all day. The picture usually wasn’t so pretty.

Tracking your time then and now is personal, it’s messy, and it’s the essence of business intelligence: collecting data and reading it for guidance in business activities that matter. Is there anything that matters more to an organization than productivity of its people? For a small office or home-based business, this might be the best BI there is.

This gets no recognition in the BI industry that I can find, at least not in the conservative world of TDWI. At least not yet.

PI — for “private intelligence” — has different issues, starting with data collection. In BI, data comes from transactions, all recorded routinely. In PI, most of it has to come from a “secretary” or from our own, tedious notation.

I dabbled in it once. The insights were good, if painful, but mostly it was tedious. A few years ago, a confluence of personal events let me do something I’d always wanted to try: hole up for a few months in a Sicilian village I knew slightly. The food was good, I had relatives nearby, and the nearby church bells rang all day and all night, four times an hour. At the same time, I had a book to edit. To stay productive, I made a game out of the work, tracking my time to the minute in Filemaker.

I liked the local food and started to hate the book, an office manual that inadvertently revealed a con game. Even so, I threw myself at it every day. But no matter how hard I tried, no full day ever resulted in more than about two hours of actual, productive work. My “quick breaks” for walks and coffee with a friend actually took up more time.

I made a Filemaker database because I could find no off-the-shelf product that would do anything close. Each period, no matter how short, had a starting and ending times I entered with buttons, and a calculation field figured the duration. A drop-down menu offered my usual activites. I could make a report for any period.

I thought some product would do that better, but I could find nothing. Then the May 2 issue of the New York Times Magazine ran an article by Gary Wolf about this, “The Data Driven Life.” My Filemaker invention wasn’t too far from what others have used, and now new devices are coming along that could make all that seem so old hat. Some people are even sharing their data on the cloud.

But as in traditional BI, the technology just gets you in the door. The show has just begun.

Most people Wolf writes about do it for personal reasons. One wanted to know how his coffee consumption helped him focus, another tried to cure his sleep apnea, and still another noticed that flax seed oil, or just lots of butter, improved his cognitive performance.

As in good BI, the experiments often raised new questions. And sometimes the new questions are unexpected, as in Wolf’s own experience.

Often, pioneering trackers struggle with feelings of being both aided and tormented by the very systems they have built. I know what this is like. I used to track my work hours, and it was a miserable process. With my spreadsheet, I inadvertently transformed myself into the mean-spirited, small-minded boss I imagined I was escaping through self- employment. Taking advantage of the explosion of self-tracking services available on the Web, I started analyzing my workday at a finer level. Every time I moved to a new activity — picked up the phone, opened a Web browser, answered e-mail — I made a couple of clicks with my mouse, which recorded the change. After a few weeks I looked at the data and marveled. My day was a patchwork of distraction, interspersed with valuable, but too rare, periods of focus. In total, the amount of uninterrupted close attention I was able to muster in a given workday was less than three hours. After I got over the humiliation, I came to see how valuable this knowledge was. The efficiency lesson was that I could gain significant benefit by extending my day at my desk by only a few minutes, as long as these minutes were well spent. But a greater lesson was that by tracking hours at my desk I was making an unnecessary concession to a worthless stereotype. Does anybody really believe that long hours at a desk are a vocational ideal? I got nothing from my tracking system until I used it as a source of critical perspective, not on my performance but on my assumptions about what was important to track.

I wish Drucker were around to respond. Wolf’s insight sounds like important stuff for everyday knowledge workers, especially those who work alone. What’s more important to a knowledge worker than time?

These experiments are often haphazard and highly personal.

Generally, when we try to change, we simply thrash about: we improvise, guess, forget our results or change the conditions without even noticing the results. Errors are possible in self-tracking and self-experiment, of course. It is easy to mistake a transient effect for a permanent one, or miss some hidden factor that is influencing your data and confounding your conclusions. But once you start gathering data, recording the dates, toggling the conditions back and forth while keeping careful records of the outcome, you gain a tremendous advantage over the normal human practice of making no valid effort whatsoever.

Yes, just as analytics gives companies a tremendous advantage over those who make less effort.

Let the BI traditionalists pooh-pooh self-tracking. The very same people might have dismissed such things as visual analysis, agile development, and at one time even business intelligence itself. Sometimes it take a few pioneers and geeks, perhaps even a secretary, to prove a concept.

When some people look at dashboards, they want to see patterns but not reasons. “They don’t want to read the fine print,” said one attendee in Lyndsay Wise’s dashboards seminar at Enterprise Data World in San Francisco yesterday. That’s what the man learned in one data-quality project for a human resources department.

He was frank enough to call drill-down “the fine print” — the suggestion that the “why?” is just noise. He slipped out before I could find out more.

Had his complacent users been victims of abusive parents or bad teachers? I’ve worked with such users. I trust them, I like them, and most businesses couldn’t do without them. But I’m curious about their incuriousness, as some of them might wonder about me.

There’s too much data, we know that. Tom Davenport ponders the overwhelmingness of it all today. The Economist reported on it last month, and Neil Raden wrote about it 15 years ago. The casual users feel it more and more.

For the overwhelmed, there’s the palliative dashboard. It works the way Mozart does for who can’t tell Mozart from Schmozart: knowing it’s Mozart makes them feel good. The palliative dashboard can be contrary to every best practice we know of and still succeed.

One person in the audience told about a pre-dashboard-era CEO who prided himself on having no high school degree. He wanted yesterday’s sales figures on his desk at 8 a.m. every morning. What decisions did he make based on that data? None! It just made him feel good, someone discovered later. Even without his reading glasses on, the patterns on the paper must have looked nice against the wood grain on the desk.

Attention dashboard makers: mind the furniture.

Everybody knows what “retention” means until they have to design a metric. Ken Rudin, once of LucidEra and now general manager of analytics at the games site Zynga, thought that he and his team could “put something together” quickly — but it actually took “four solid weeks of discussion and debate.”

About 50 million people play Zynga games every day. It’s the leading online social gaming platform, according to Ken, and it’s grown from zero in 2007 to revenues of “a few” hundred million dollars annual revenue. Every day, the company captures 20 to 30 billion records of data, and Ken and his team use that data to improve revenue, viral marketing — and customer retention.

Zynga players play free. The revenue comes in a few dollars at a time for “virtual goods.” In the popular game FarmVille, for example, a player might get tired of the old-fashioned plow. The tractor upgrade costs $2.

“There are tons of different ways you can think about retention,” he laughs, “and which one should we use?”

How do you know when a customer has left? “Unless we don’t get a note saying, ‘Hi, we’re no longer playing,’ how do we know?”

Of course, no player’s going to make it that easy, so how long should Zynga wait before considering the player gone? A week? A man could have dropped his virtual pitchfork for a real vacation — or he could have plowed the last row.

Ken dealt with analytics all the time at LucidEra, but games were new to him. He’s learned a few things.

“It turns out, as you might imagine, that it depends on the game,” he says. The average simulation-game player tends to visit frequently, for example. Poker players, though, are much more likely to come back after, say, a three-month gap.

The retention curve also varies by the length of each player’s tenure. A new player who stays away 30 days is much less likely to return than a player who’s been at Zynga for years. Ken now puts users in three basic tenure buckets: “new,” “mature,” and “elder.”

Whatever question you try to answer, it has to be actionable. “There are metrics, and there are metrics that matter,” he says. If volume plunges, were the missing players mostly new ones? If so, it could indicate frustration; perhaps the games need better tutorials or less functionality at the beginning. Or were most of the missing the long-term customers? If so, perhaps the games haven’t offered enough challenge.

Ken expects growth when the economy improves. “When we look at what happens over holidays, such as July Fourth and Thanksgiving, usage really drops. Then it picks up as people go back to work,” he says. “[The games] are part of their routine. On vacation, players break their routines. They sleep late and spend more time with family. They don’t play the game.”

“It’s fascinating,” says Ken. “In analytics, so much of the problem is figuring out what the question really is.”

I think he means that it’s a great game.

I like the sound of Government 2.0: Collaborate with citizens online and you can change government from a sewer-dwelling raccoon into a purring housecat.

Social media lets us try for a kind of politics that was impossible until now. I hope for great results. For many, Government 2.0, or “collaborative government,” will mean just “friending” a local cop. But in full flower, Government 2.0 can mean far better service, and far more government-and-citizen collaboration than ever before.

Even before we had social media, the glare of public attention was a proven antidote for bad politics. Citizens getting up their elbows in policymaking has always been another strong medicine.

Trouble is, that “sewer-dwelling raccoon” is always smarter than people think. When he’s hungry, he purrs like a housecat and covers stinky laws with high-minded names. Advertising fools just enough voters — so often complacent and impatient — to throw a new law onto the books. On and on it goes.

Such a stinky new law is just what Californians got in 2000. Proposition 34 was sold to voters as campaign finance reform. It was a ruse. (A few days ago, a report confirmed suspicions, and a major drafter of the proposition insisted he was “outraged.” Yeah, and round up the usual suspects.)

One other fix, more honest, came 100 years ago: California amended its constitution to give citizens the ballot proposition. It was the only way for voters to bypass the paralyzed Legislature and loosen the Southern Pacific Railroad’s grip. It worked. But more recently, ballot propositions have helped tie the state’s budget in knots.

In the long run, who knows how social media, visual analysis, and other tools may be used in government? What will matter most of all is who uses them. If it’s “the people,” which people?

I hope this new, pervasive politics mobilizes a new wave of smart activists — the way desktop publishing and, later, weblogs enabled editors and writers. Or the way tools like Tableau and Lyza are enabling independent-minded, creative analysts today.

As these activists learn about politics, I also hope that more citizens than ever before step up to watch, push, and verify. Such a voter would be Citizen 2.0, the real hope.

Otherwise, it’s going to be that raccoon again — this time on Twitter.

What’s the hardest part of training a new data analyst? Resetting the trainee’s mindset.

“They start out with the idea that there’s a right answer,” says Joe Mako.

Joe’s leaving his job — where about one year ago he began analyzing data — to go work for the producer of Lyza. Lyzasoft CEO Scott Davis sees him as a “prototype” of a kind of creative, resourceful analyst that Lyza was designed for. Joe will engage with other analysts to evangelize Lyza and to help new users ease into the flow.

Joe, 29 and a veteran of two Army tours in Iraq, started out on the help desk. He answered calls from within the company, an ISP. Many callers couldn’t or wouldn’t analyze their own data, so Joe did it for them. His boss also enlisted his help — and now won’t dare go without a backup.

The first people he’ll help get into the flow are the two women who’re replacing him, and he’s got to do before he starts at Lyzasoft on November 9. They’re some of only a few in the his group who applied. Most others refused the “boring” work with “ugly” data.

New users, he says, want to know, “Where’s my wizard?” There is none. “But that’s why I enjoy these tools.” He uses Lyza and Tableau primarily. “They stay out of my way. They enable me. It’s just me and the data. … That’s what’s neat. But [new users] don’t know where to start.”

“I’m handed crazy files without any structure,” he says. The first thing new users have to know is that, no matter how ugly the data may be, it really can be cleaned up. He demonstrated to his new trainees, he says, and “they were blown away.” After that, he started showing them how they can clean up data on their own.

He explained basic steps and functions. Then he showed them how to combine tools, such as how to use two functions in sequence. And deeper still.

“It takes time playing to figure out where you need to get to,” he says. “You have to just go and play. If one thing doesn’t work, you try something else.”

“I always thought that data was exact,” he says. “If not, it was garbage and I’d throw it out.” But he later learned that there’s usually only a portion that’s garbage — that somewhere within the crazy mess there’s a story. “Even if every data point is wrong, there still might be some trend you can see. If there’s a bunch of ugly data, how do you figure what he story is?” It takes a willingness to figure it out, to untangle it, to find out what’s in there.

That’s a skill, not a talent, he says. “I’ve watched [his two replacements] get it closer and closer, learning to merge other data in, to reshape it and finally produce the output.”

Closer and closer. Business will trudge ahead, training a Joe here and a Joe there until people don’t complain anymore about boring work with ugly data. Someday, many more people will welcome the chance to do this work.

With all that email that piles in for vi*gra and unlucky Nigerian princes, we assume that someone, somewhere, makes tons of money on it all. But some stealthy University of California researchers at Berkeley and San Diego concluded that spammers may be easier to thwart than we thought.

They tell the story in “Spamalytics: An Empirical Analysis of Spam Marketing Conversion,” published about one year ago. (Download the PDF here.)

To gather data, they devised trickery of their own: They infiltrated an existing spam botnet.

By infiltrating the botnet parasitically, we convinced it to modify a subset of the spam it already sends, thereby directing any interested recipients to Web sites under our control.

The team studied three campaigns: one selling pharmaceuticals and two propagating malware. They tracked nearly a half billion spam emails to count successful deliveries to mail servers, successful passes through anti-spam defenses, user visits to advertised sites, and sales and infections. Throughout, researchers were careful to avoid doing any harm; users responding to infiltrated bots could never actually buy drugs or download malware.

In the study’s small sample, only about a quarter of the all spam leaving their cave ever reached a mail server. Only about 16 percent reached users’ inboxes that targeted Hotmail, Gmail, Yahoo, or Barracuda. Only about one in about 12,500,000 users originally targeted ever took the bait — a 0.00001 percent conversation rate.

Researchers extrapolated from their tiny sample — roughly 1.5 percent of all traffic on this network — that the spammers might gross about $3.5 million per year, and higher if users come back for more.

But low conversion is what we’ve always assumed. What’s critical, and tricky, is estimating costs. The researchers figured that even at $80 per million, an average derived from anecdotal evidence, costs would be too great unless the spammers were vertically integrated — the vi*gra and the spam delivery all operates under one roof.

This discovery is “heartening,” says the paper.

“… profitable spam campaigns require organizations that can assemble complete “soup-to-nuts” teams. Put another way, the profit margin for spam (at least for this one pharmacy campaign) may be meager enough that spammers must be sensitive to the details of how their campaigns are run and are economically susceptible to new defenses.”

Spam may be easier to beat than we thought. Anyone for a “surge”?