datadoodle

Text analytics was one of those things I heard about every so often. Like so many terms in this business, the term comes out of a speaker’s mouth or PR person’s press release only to blow away. There’s no story, no context, nothing to chew on.

Then came a press release at BI This Week with a rare combination: surprise and concreteness. It said text analytics would help with food safety. I’m all for food, but I had no idea what text analytics had to do with it.

I emailed UK-based Linguamatics, publisher of the nifty tool they call I2E. What’s this I hear about food? Product manager Phil Hastings, ready to call it a day in Croatia, called to explain the features to me, barely post-breakfast and not fully verbal. I2E was indeed a powerful little thing, but I still didn’t get the food angle.

It wasn’t until I got William Hayes on the phone that things started making sense. He’s director of library and literature informatics at pharmaceutical research company Biogen Idec. They don’t do food, but close enough.

If you think the Sunday New York Times is enough for one day, consider what the research community has to bear. Hayes says, “If you’ve got 20 million articles to read, where do you start?’

“The research industry works under a tougher knowledge model than terrorist intelligence gathering,” says Hayes. “Our ability to tap that ocean of literature is like dropping a line into the ocean for fish.”

In general, a scientist can read 150 to 200 full text journal articles a year, he explains. A curator can review about 100 abstracts a day “for a few days before you start going nuts.” Text mining is the only way to keep up with the ocean of literature produced each year.

The food industry fries potatoes, but it also has to keep a lookout on research.

TNO information analyst Fred van de Brug told me the acrylamide story: Most people in the food industry missed the first warning. Scientists had published a discovery in 2000 about a possible carcinogen known as acrylamide, which can develop in starch-rich foods like potatoes as they are fried. By the time the warning finally hit the public media in 2002, millions of people became frightened, perhaps unnecessarily. Text mining would have given food processors time to head off a crisis.

I2E is more agile than standard text mining. You can learn to use it in a few hours. Hayes told me, “If you can remember bits of grammar and have some concept of what you’re researching, it’s a piece of cake.”

It’s a story in progress for BI This Week.

Tags: BI This Week, food, text analytics

Evidence came in yesterday afternoon that, so far, BI is doing well in the new economy. Greg Turman, director of sales for the Western Region at Corda, phoned to say, “I’m going through the roof,” by which he means sales are good. “I’m seeing tremendous startup activity. People are saying, ‘I can no longer guess. I’ve got to know what’s happening so in case things get tight I’ll survive.’”

What’s behind the surge? He figures that people have finally understood the value of analytics and visual analysis.

Tags: Corda, economy, infovis

DM Radio editor Eric Kavanagh puts on a scary mask for a special Halloween show this afternoon: “Scary Stories of Information Management.” Scaring you will be quite a trick after a year of cadaveric prose in BI articles and blogs. But there’s probably more where that came from. He wants your stories of fright and demons. Details here.

Tags: DM Radio, events, stories

In defense of the lively, Mark Madsen observes the nature of resistance to open source BI tools. An excerpt:

Overcoming someone’s resistance to open source in your organization means that you probably need to educate them, given that they use open source every day without thinking about it. It’s in everything from cars to cell phones, as well as almost all the commercial BI tools shipping today. More likely, they are resistant because they (a) are threatened in some way by the change you propose, (b) face organizational obstacles like educating the legal department about licenses or (c) face political consequences you aren’t aware of. It’s often their personal situation that is the biggest factor, given that most objections are easily refuted as myths.

Tags: Mark Madsen, open source

It’s hard to see through the smoke as our financial house burns down, I know. But what I’ve noticed is more interesting: the first signs of rebuilding.

This month, three experts I read—visual analytics expert Stephen Few, Competing on Analytics author Tom Davenport and digital-media economy specialist Umair Haque—all seem to have knit recent blog posts with the same thread: honest value in business and the economy.

Continue Reading »

Tags: economy, recession, Stephen Few, Tom Davenport

“Black swans” are the anti-gravity of predictive analytics. These events are so far off the charts that we dismiss the possibility out of hand. But when one occurs, it’s a doozy.

The Panic of ‘08 may lead us straight into one of these, says Nassim Nicholas Taleb, author of The Black Swan and Fooled by Randomness among other works. He and his mentor, mathematician Benoit Mandelbrot, told NewsHour’s Paul Solman last week that what’s coming might make the Great Depression and the Long Depression of the 1870s seem small.

But before you lose sleep, mind this caveat: Taleb himself calls predictions “bullshit.” In a talk he gave in 2006 he said, “We’re suckers for anyone who talk to us about the future.” (Listen here.)

He’s getting lots of attention these days. Taleb foresaw this financial crisis. Lately, he has made money for investment clients based on that prescient gloom. Others saw it coming, too. A banker told me in 2003 when I asked for his outlook, “Oh, pal! You’re talkin’ to Darth Vader,” and he sketched the scenario that began unfolding last year.

It’s all about turbulence, Taleb theorizes. Our system is “over-optimized” and has too little slack. It can’t absorb much shock. Taleb says that’s how a small shortage of oil sent the per-barrel price from $25 to $150. It’s also how a small mistake in one big bank sends shivers through other banks.

Perhaps it’s also how a dollar bill on a Wall Street trading floor—imagined last week by The Onion—caused a surge of optimism.

The Onion’s jokes often come true, but optimism is out for now. Today the world feels like it did after September 11, when we assumed the terrorists had another one coming at us. Or after the 1989 earthquake in San Francisco. I had grown up in the area and had felt lots of quakes, but a year my stomach clenched whenever a heavy truck rumbled by.

If there’s any comfort, it might be in what the ever-entertaining Ed Brown said once in a talk at Tassajara Zen Mountain Center. He told about a middle-aged man he’d seen at the baths, submerged in the hottest water. A few kids watched him as he encouraged them to try it. The man said, “Don’t resist the heat. You just kind of let it enter you.” Ignore the voice in your head that screams at the unknown.

While there’s time, read Taleb’s Fooled by Randomness. I haven’t yet read The Black Swan, but I’ve heard it’s also very interesting. Available now.

Oh, and take off your scary mask when you walk through any financial district on Halloween.

Tags: Panic of '08, predictions, predictive analytics

A recession would benefit business intelligence, say two industry experts I talked to last week.

Continue Reading »

Tags: Dave Wells, recession, Tableau, TDWI

Last January, I surveyed BI consultants to see what the season’s recession was doing to BI. Things were going fine, most reported. This week I’m following up with them on the Panic of ‘08.

Continue Reading »

Tags: BI This Week, Mark Albala, Panic of '08, predictions, recession

Restless minds will want to know what Asian manufacture of furniture, clothes, electronics and other goods has to do with business intelligence.

A globe-trotting industrial engineer who’d rather not be named has been telling me about different perceptions of quality among nationalities. He works on contract to American companies to ensure that product quality lives up to agreements.

When Americans buys new stuff, they assume it’ll come out of the box without dings, dents, scrapes or other flaws. Seams will be tight, electrical joints will be well soldered, paint on the fender will match the hood, wood veneer will be smooth.

According to the engineer, Chinese and Indian manufacturing and warehousing staff he’s worked with see it differently. When he flags a wooden cabinet, for example, with a deep gouge on the corner, the Chinese warehouse manager shrugs. “He’ll say, ‘That’s just because it was moved around the warehouse.’ It’s nothing to him.” Same with a chair with one shade of fabric on the armrest and another on the seat.

What about data? If that’s their cultural bias about dings in furniture, how do they feel about dings in data? Is carelessly handled data as easy to detect?

Tags: culture, IT

In a good example of “show, don’t tell,” Tableau Software’s weblog demonstrates the power of its product with a story: how rich, middle-income and poor voters compare in liberal, conservative and battleground states. The political story is awkward to tell in words, but it’s easy in pictures. Pictures that tell stories is what Tableau’s all about.

Continue Reading »

Tags: marketers, stories, Tableau