This afternoon, I finally closed the long-running survey of “those who analyze data.” The results are seeping in to Datadoodle headquarters. I’ll release them in stages over the next two months: first, highlights, then more highlights, and finally a preview report and a final report.

I opened it in mid-February this year. It has 221 responses. I’d say that’s a good number from a small platform like this weblog in this survey-saturated industry.

In case you didn’t realize it, I’m no data analyst myself. My stint in market research that ended more than 10 years ago entailed little data grooming, reshaping, or cleansing. Someone did it for me, and only then would I paw through it. But today, it’s all up to me — me and my handful of data-savvy volunteers.

I’ll identify and thank each one publicly as things progress, along with a few people who helped promote the survey.

The very first look will probably come in my columns at BI This Week (TDWI) and Information Management. After that, I’ll issue a report, first in a preview edition for respondents who asked for it and a few others.

Just watch this space for links.

As if to renounce one more convention of business, a Berkeley-area businessperson I know couldn’t find data, so he went out and got some of his own. He had to evaluate market areas.

For three days, he stood by an ironing board with a map on top in front of the Cheese Board in Berkeley’s Gourmet Ghetto. Each customer coming in or out got handed two stick-on dots, red to mark home and blue to mark work. His map quickly revealed patterns and commute routes.

It was easy. “Everybody’s curious,” said Terry Baird, who had made a career of cooperative/collective food retailing. He had sold advertising for 10 years, and later found himself appointed to unravel the bankruptcy of a once-huge food distributor, an experience he calls “my MBA.” He approached it all without training, only logic and nerve, such as with the ironing board. “It’s like running a three-card monte. A group gathers around you.”

In 1996, he was part of a newly formed collective inspired by the Cheese Board. The new collective trained there as they decided on the new store’s location.

Terry’s question: would any of the proposed locations for a new store overlap another store’s market area?

In three days, he had the answer. On weekdays, none overlapped. But on Saturdays, all did; that’s when people drove in from all over the Bay Area.

The best software might be the kind made without starch.

Search all the business literature you can and you’ll never find data analysis compared to romantic love. But, hey, why not? Love’s trajectories might hint at what the business world’s newly enabled generation of data analysts can expect.

These data analysts tend to be independent, are often creative and at least partly self-trained. They’re strapped to rockets from Tableau, Lyzasoft, Predixion, and others, tools that are at first deceptively toy-like. Aren’t they analogous to the garden variety teenager? Bothg groups revel in newly discovered tools, while both pursuits are fundamentally social — as Lyzasoft CEO Scott Davis observes about data analysis. His blog post got me thinking about this.

Everyone shows up ready to rumble. They’re fascinated with the possibilities, they experiment in private, later they have a blush of quick results followed by a long trail of self-training on finer points.

Each group’s toolset is potent and designed for early success but never early mastery. They make lots of mistakes. In love and analysis, people fall for the wrong data, mess up good data and dates, do all kinds of things they wish they hadn’t.

Without realizing, they face danger. I’ve noticed that behind most good trends comes a rotten sibling right behind it. Think of the history of other social events: Hippies begat the Summer of Love and then came Altamont. We celebrated “free love” and then came a surge of sexually transmitted diseases. Baseball begat the World Series and then came batters on steroids. PageMaker begat self-publishing but then came the ugliest lost-cat posters ever tacked on a telephone pole.

You may already wish that bad analysis would go away. Pete Warden, for one, warns of some fabulous ways people trip over new data. We could easily call this stuff “data porn” and ignore it.

But there are even more treacherous pitfalls. These potent tools can change everything in a flash (at the “speed of thought”). One minute you’re in orbit, and the next minute you wish you were dead. With sex comes the hazard of a painful breakup, and with data analysis comes the danger of unwanted speech that’s too hot for any public platform. Oops!

We have ways to deal with all that, but it’s never pleasant. The rejected lover picks up and leaves, and the analyst just finds his creative viz zapped off the cloud — by those who are themselves learning a new role.

The lover and the analyst both feel hurt, perhaps betrayed. Wasn’t each playing by the rules? Wasn’t each part of the group? Suddenly each one feels rejected for reasons that a hasty explanation doesn’t quite calm the hurt feelings.

In hindsight, we realize we shouldn’t have been surprised. Social pursuits can be like this.

By the way, who said good tools were the end of the story? Well, most vendors did. Some teenagers think so, too. But even slightly more advanced users know that technical proficiency is only the price of entry. We do the real work in many long conversations and collaborations with words, data, gestures, misunderstandings and reconciliations, and on and on.

Here the analogy breaks. The tools will keep getting better while the bodies fall apart. But the lesson’s the same: Tools enable, but conversation — better known in the business world as collaboration — is really at the heart of our work.

When you come face to face with unfamiliar data, how do you proceed? How do you avoid sending you and your shiny “speed of thought” tool slamming into a dead end? Dan Murray’s got a routine — and he’s also got certain music and right-brained books to go along.

Dan’s first rule: “Don’t pre-think.” It’s the hardest thing for people to learn, he says. “If you go into [data analysis] thinking you know where you’re going, you easily miss the granule of gold.”

He’s the chief operating officer and heavy-hitting data analyst at InterWorks, Inc., an Atlanta-area business consultancy. What seems to me like an unending stream of mid-size businesses from all different industries has kept him running days, nights, and weekends to make sense of each one’s data and unravel old data knots.

From an airport somewhere in the South, he explains, “You have to think like a writer thinks. You don’t know where the story’s going to go.” Screenwriters and novelists often say in interviews that their characters veered off in directions the writer hadn’t anticipated.

He’s been analyzing data ever since spreadsheets first became available in the early ’80s. “I was a huge spreadsheet guy.” Now his tool of choice is Tableau.

The routine goes something like this.

First, get the big picture. Grasp the general outline. How many records do you have? What’s the highest and lowest? For example, if you’re looking at a company’s sales, how many sales, units sold, and so on?

Look for what pops out. Trends often make themselves obvious right away.

Find groups. Build a bar chart to see how it all breaks down. If you’re looking at sales, make groups of products, divisions, for example.

Lay out timelines. Build time series to see any long term trends. Start simply with years, then break it into more detail.

Make maps. If the data contains locations, throw it on a map and see what clusters appear.

Go on tangents. Try making some measures into dimensions. For example, if you have a million invoices, with a range of up to a million dollars, where do most invoices fall? Try cycling through every type of chart. Remember, the cost of any view is just one click.

Look into outliers. Outliers may be just bad data, or they may be interesting. A good place to find them is in scatterplots. “Most of my interesting discoveries are in scatterplots,” says Dan. Seemingly unrelated numbers sometimes have some kind of interesting correlation.

Combine. Put all the charts done so far into one dashboard. Filter all the views based on [things I highlight]. There you can see it all at once. Brains don’t remember more than one or two things at one time, but here you see it all together.

Repeat. Good tools make false steps easy to back out of.

Keep an open mind. He plays music, often the piano music of Frank Kimbrough, such as”The Spins.” He emails, “The lyrical and circular notions of this song reflect how I do analysis. He circles, he dives, he riffs, and then he comes back and does it again in a slightly different way.”

Present and persuade. Jazz, right-brain thinking, motivation, surprise, discovery — it all results in discoveries that must be communicated persuasively for any value to result. Dan recommends the two books by Dan and Chip Heath, Made to Stick and Switch.

Three hours of analysis will show you plenty. “You’ll know just as much as the insiders know.”

Do you have a routine for analyzing unfamiliar data? I’d especially like to hear from users of many different tools, from the most advanced to pencil-and-paper. Please introduce yourself here.

So many people who should know better seem to miss the point when they mention Tableau. Why? I asked BI veteran Stephen McDaniel for his thoughts — which he gave, but then went on to suggest an almost unheard of challenge: a data analysis face-off among vendors.

Consider this description by a BI analyst: “Tableau provides business analysts speed of thought visual analysis on data held in memory on their desktop machines.” All that’s fine, but it may as well have been about a whole bunch of other tools, too.

At the root of this fuzz, explained McDaniel, is that most analysts who concern themselves with tools don’t actually use the tools. They rely on demos , marketing, and hearsay.

Though much of McDaniel’s recent work has centered on Tableau — his second book is Rapid Graphs with Tableau Software and he gives training sessions around the country — he also has a long, credible trail back through BI and data mining. He was director of analytics at Netflix, and has worked with more than 50 companies in BI. His first book was SAS for Dummies.

“I love SAS,” he says. Still, he remembers his sister in law’s reaction to his book on SAS. She was not an analyst but a “people manager.” These are the ones, he says, who have hated BI because “it had been made into a priesthood.” When she had looked through the book, she said, “Oh, this is great” and put it down. But she read the Tableau book for a half hour and said, “You should come talk to some people I work with.” She had recognized what she could do with the tool.

McDaniel’s sister in law and many like her don’t care whether the data is “in memory,” they don’t see themselves as business analysts, they take “desktop” for granted, and they know “speed of thought” is just gloss.

The list of features really doesn’t matter. All that really matters is whether someone can do what needs to be done with the tool.

McDaniel imagines a throw down, a data analysis match. It would be open to any BI vendor. Each vendor would send their best people, and each team would receive a uniform set of data. Over some defined period, teams would analyze and then present the results to a panel of vendor-neutral judges.

The reward? Perhaps a signed copy of a Stephen McDaniel book, or maybe a beer, possibly both. But certainly, repute.

What do you think of the face-off idea? Please write a comment.

It was International Workers’ Day on Saturday and the official release day of Lyzasoft’s latest product: its foray into “free.” It’s a good way to say “power to the people.”

Some people associate that slogan with protests and even violence. But I think the best paths to power usually involve well-analyzed data, whether in public life or at work. Now the Little Guy has a potent new tool to deploy.

Lyzasoft founder Scott Davis calls Lyza on Lyza Commons “the YouTube of data.” This fully functional cloud-based version of Lyza is a strong tool for office-based, home-based, cubbyhole-based, dorm-based, or public library wifi-based users and groups. Import your data from whatever sources you have, refine it, share it with to whomever you like, and even charge toll over Paypal if you want to.

“Obviously,” Scott says, “what we’re doing is saying, ‘This thing can scale.’ But instead of going for the uber-enterprise as our leading play, we’re saying that what’s unique about this technology is it can make it to everybody within a small and medium business without having to have a big IT team around.”

Lyzasoft’s second, paid tier serves customers who need private clouds. That version starts at “small” for $150 a month, seating up to 10 users and providing “plenty” of storage. Go upward through “medium” and into “large,” and you pay $2500 a month for up to 200 users.

Wait, you say. You’ve heard this “YouTube of data” thing before. Yes, just three months ago another YouTube of data launched, Tableau Public. (I wrote about it here.) Tableau, Lyza, and YouTube itself all say “power to the people” by popularizing a medium with free, easy-to-use tools and a venue. Each one’s growing crowd of Little Guys and their audiences turns into a movement that the those in executive suites can’t help but notice. At some point, YouTube and those who follow its model hope that “free” leads enough customers to “ka-ching” to yield a profit.

YouTube seems to be well on the way. Its ready-to-roll movie theater had fired the imaginations in a waiting mob. These filmmakers-to-be had been trained over years of TV and movies to understand film and crave a chance to do their own.

Is there a waiting mob of would-be data analysts? One pioneer of free analytics is skeptical. LucidEra founder Ken Rudin, now vice president of analytics at Zynga, says you need more than free tools, no matter how easy the tools are to use. He says, “Tools are only as valuable as the questions you ask.” One of his biggest hurdles was getting customers to appreciate the possibilities of analytics.

But the YouTube idea is more than tools. It’s a game and a self-reinforcing mob. The tiny films YouTube users make don’t just play as if on a jukebox, they’re scored, they’re answered, and commented on. It’s like the difference between voting in a little booth and going out on a street march. It reinforces and stimulates. Unlike most business environments, it asks people to play, which is how Lyza Commons and Tableau Public users will break out into creative and incisive data analysis.

I also hope there’s a new supply of analysts. Ken Rudin and others are hungry for them. (In fact, if you’re a data analyst and you want to work with cutting-edge technology and data in one of the world’s largest databases, email Ken today at krudin@zynga.com.)

Power to the data analysts!

One of the pleas in Mark Madsen’s fascinating keynote at the TDWI conference in Las Vegas was to let the “Barts” work. The Barts are, of course, the Bart Simpsons among us, the sometimes nerdy rebels who actually come up with interesting analyses and other useful things.

When the lights went up, a “Bart” was right nearby me at the big round table. Though he “loved” Mark’s salute to his work, consultant Rick Paul wanted more.

As a Bart, he works with a lot of “Marges” and “Homers.” In Mark’s model, the Homers are the everyday business intelligence consumers, about 80 percent of most work groups. The Marges are about 18 percent, and they actually think a little. The last 2 percent are the Barts, the ones who analyze and invent — and who’re limited by the BI systems built for Marge and Homer.

The more painful obstacle facing many Barts, says Rick, isn’t about any technology.

He tells how his team started with three people, all data architects, all smart. “We could do anything,” he recalls. Now the team has 120 members, many of them Homers. They’re of the 80 percent who consume but don’t invent or even think very much. “They’ll fake inability,” he says, “to tempt or coerce the three innovators to do their work. They say, ‘Oh, you’re so good at this. It’ll just take you a few minutes to do this.’” It really does take only a few minutes, but “it’s not thinking work.”

“I’m lazy,” he says, “but when I don’t want to do something, I figure out how to automate it.”

Rick says he’s still trying to figure out how this situation can be resolved. He mentions isolation, but he also thinks of encouraging the 80 Percenters to have some vision for their own careers. They should have some way to “add intelligence to their own work on a daily basis. They should be actively engaged with their work.”

Smaller teams might also work, he says. In a team of 120, it’s easy enough to do nothing for weeks at a time. It’s much harder in a team of, say, seven members.

“The innovators have to be positioned to influence the company,” he says, “but not be abused.”

Why do creative data analysts meet resistance? I’ve written about them before: many are young, they’re smart, and their credentials might still be slim. But they’re still a value to their employers, no matter who knows it yet.

At least some of it’s about Boomers vs. the up-and-comers. One corporate communications consultant I talked to, Liz Guthridge at Connect ConsultingGroup, has seen all this in action.

“Boomers are still interested in experts,” she says, “people with loads of experience and loads of credentials.”

Liz tells of one client, a Syracuse University alum, who hired a young guy as CIO. The newly hired young man was “impressive,” Liz recalls. Among other attributes, he had done a double major in college of public relations and information technology. This was his first job out of school.

The client, she says, kept belittling his knowledge — even though he was, Liz says, “really solid.” The problem was in the way the new CIO communicated with the boss.

Liz has advice for younger people with important things to tell their Boomer bosses. The first three points:

1. Think of the exchange from the senior person’s point of view. Since boomers love experts, always provide proof points. And not from Twitter but from boomer-recognized sources like the Harvard Business Review.

2. Network from the bottom up. Find out who the senior person listens to and talk to them. No explicit appeals, just talk.

3. Use the senior’s preferred communication channel. The woman who hired the CIO liked to schedule phone calls, not email or text.

The young CIO, she tells me, is still on the job, though he’s cut back his hours.

Lyzasoft wasn’t among the 38 exhibitors in TDWI’s Las Vegas exhibit hall. Lyzasoft sponsored no part of the lunch, and they hired no stage magician. But their buzz was the loudest I heard over the event’s five days.

Others may have heard different buzz because buzz varies. Business intelligence elites gather every year at TDWI’s big Las Vegas event to teach, and they end up schmoozing, too. Over beer, food, and sometimes playing cards, they compare notes.

Is anyone seeking a consensus? I suppose someone might, but the interesting ones just play with ideas, reflect on what others say, make a joke, and think about it. If there’s any “truth,” it develops during a lot of talk and thought, whether it’s about politics, tofu, the future of passenger rail in America, or business. That goes for any kind of conversation, whether the medium is words or data.

In business, the conversation is somehow forgotten in favor of the data. But to Scott Davis, CEO of Lyzasoft, the conversation is critical to understanding the data. “A chart has no context at all,” he said in mid February. “The conversation is what’s really valuable.”

The conversation-free, top-down “single version of the truth” isn’t always useful for those who need to manage data for specific uses and contexts. Its “truth” may in fact be no better than Soviet planners’ forecasts of market demand for women’s lingerie. “A single version of the truth,” said Third Nature research director Mark Madsen in Las Vegas, “is true for a single beat of the corporate heart.”

Enter Lyza 2, Lyzasoft’s new version of its data-wrangling and collaboration tool made for data analysts determined to create truth for specific uses and context. The first edition of Lyza offered Excel-like personalization. In the new edition, collaboration seems to have been the guide.

You could see this year’s improvements coming in last year’s email from Lyzasoft CEO Scott Davis: “Even though they are quants, their world is personal,” he wrote. “Relationships are vital. They think in terms of ‘who do I know who knows X type of information sources?’” He could have also been talking about journalists, artists, and anyone else who has to hear signals within noise.

In the new edition, Lyza encourages fluid interactions with a variety of social-media tools: email, Twitter-like messaging, SMS messaging, bookmark collections with annotations, and other tools track and fortify discussion. Lyza lets people work easily with other smart people they trust. If “Steve” believes that “Brian’s” work is good and “George’s” work is not, he can work with only Brian’s data. It also publishes to the new tool, Lyza Commons, for even greater collaboration while retaining users’ ability to interact with data. Lyza 2 loves a good conversation.

The data and everything that happens to it gets tracked automatically. Unlike in Excel worksheets, changes are transparent. Automatic documenting allows any change to be dug up and fixed. If only the data-free conversations in politics and other parts of business had such a tool.

I was surprised to hear spontaneous praise for Lyza’s new version. eLearningCurve education director Dave Wells and Third Nature principal and one of the event’s keynote speakers Mark Madsen both did. I heard the same from several other BI experts, too. Madsen even gave a brief look at Lyza in his Executive Summit presentation on the future of BI.

I harmonize with people who appreciate Lyza at least partly because I think it’s smart to let people work the way they want to work — the way people have always worked. They prefer working with people they trust and with tools that respond. Everything else is static.

I’m sorry to tell you serious types out there, but visual analysis is often a game — in fact, one of the best games in town with Tableau Software’s visual analysis tool. Now Tableau Public is going to bring it to the masses.

In the same way that YouTube spawned a surge of new filmmakers, Tableau Public — free, running the same engine as its desktop sibling, and embedable — will bring on a new generation of data players and spectators.

I was a spectator at a data visualization conference one afternoon two years ago. Tableau Software director of visual analysis Jock Mackinlay had finished his presentation and another person had started his. Yet someone at the control board forgot to flip a switch, and Jock’s live screen remained on one of the room’s big screens. Jock assumed his screen had been hidden, and he kept playing with the data. I don’t have to tell you who seemed to have the audience’s attention until someone pointed out the problem.

The mere visual distraction was minor. Even without narration, I got caught up in the apparent drama as he tried one look at the data after another.

Not long after that, I wondered aloud to someone at Tableau about data hobbyists. I imagined people who foraged for data to analyze then publicize it to start conversations, collaboration, or duels. Data would be their raw material of choice just as scrap metal is to some sculptors or overheard conversations is to some fiction writers.

There was no such community visible then. But I realized this week that I know one now: Dan Murray, a skilled, dedicated Tableau user. He jokes that he’s a “freak” because he analyzes data from the federal budget and posts his often provocative analyses. He’s already been answered by at least one who disagrees with him.

In beta and since its February 11 launch, Tableau Public has hosted a flurry of visualizations, including these: a map of top venture capital firms investments by U.S. region; a chart showing how long it takes to build a technology empire; a history of earthquakes in Haiti; a neighborhood breakdown of housing supply in Seattle; trends in U.S. high school graduation; and studies of deprivation and marginalization in education. In most cases, spectators can become players by selecting subsets of the data to find answers to their own questions.

With popularity comes some misuse. Many of the charts will break rules, such as what happens in another kind of game, YouTube. A New York film editor I know complains that many YouTube-acculturated film editors have neglected basic editing principles. She writes that they rely so much on special effects that they “can’t put two shots together and have them work as an unembellished edit.” On Tableau Public, there will be pie charts, chart junk, and even baselines that do not start at zero. We’ll survive it.

But what’s all this got to do with the very serious practice of business intelligence?

Like monks must have done when printing presses began producing books for the masses, many priests of business intelligence will stand aside, arms folded in the aspe chapel. But I predict that before long even they will appreciate a wider, deeper pool of analytical talent ripening for training and employment.

I suspect that the new bunch will have been sharpened by the give and take of public exposition. They’ll also learn from playing in a huge community the way artists and craftspeople of all kinds improve their skills when they bump into peers every day.

This is a new clue for the future of BI. It can’t help but improve data analysis in business. So let the games begin.