There are the dream organizations that deploy data analysts wisely. Then there are the nightmares, such as the I.R.S. as portrayed in David Foster Wallace’s last novel, The Pale King, reviewed yesterday in the New York Times.

… In a universe of veiled and veiling numbers, the task of drawing the true [data] out into the light and holding them up for inspection, clear and remainder-­less, really is a sacred one. … The problem, as I.R.S. recruits soon discover, is that neither moral nor heroic codes hold true anymore.

These recruits work with “excruciating difficulty … in an age of data saturation.”

The [instructor] presents “the world and reality as already essentially penetrated and formed, the real world’s constituent info generated . . . now a meaningful choice lay in herding, corralling and organizing that torrential flow of info.”

One character is the data psychic, Sylvanshine, who can “glean trivia about anyone simply by looking at him.” But, as if to prove that good data is far from the end of the story, he has a problem.

[He] is “weak or defective in the area of will.” Nor, due to endless digressions, can he complete anything. No one can; in “The Pale King,” nothing ever fully happens. That this is to a large extent a metaphor … becomes glaringly obvious when we hear one unnamed character describe the play he’s writing, in which a character sits at a desk, doing nothing; after the audience has left, he will do something — what that “something” is, though, the play’s author hasn’t worked out yet.

Let’s see, will an “easy to use,” “speed of thought” tool help? Is there a tool for Sylvanshine and the others?

No, at least not until the next update. But this is why business intelligence is fascinating. Under cover of tools and data, we touch the heart — throbbing or dead — of the organization.

Self tracking for performance has a place on the map now thanks to the May 2 New York Times Magazine article by Gary Wolf. But along with praise and interest, “The Data-Driven Life” also drew harsh, skeptical reactions.

Many of the objections were of the “if man were meant to fly, he’d have wings” variety. But many others were valid.

The practice will run over a few bumps before it joins mainstream performance management and business intelligence. Unlike the impersonal data we know and love, keeping data about oneself can be uncomfortable, difficult, and downright weird.

One of the articulate skeptics called it “robot envy.” In his weblog, Marginal Utility, Rob Horning summed up his objections in the final paragraph.

Numbers can provide only one sort of “truth” about ourselves, and to pursue it we must surrender or compromise other kinds of truth—for example, the intuitive faith we have in our qualitative assessments of our dasein. [...] In other words, we give up our soul for a spreadsheet.

I’d like to meet the spreadsheet that steals souls. Until then, I’ll cling to my belief that no spreadsheet, not even Excel, has any more power to do that than a blood pressure cuff or a bathroom scale.

A more credible response came on the New York Times site from “Matt” in California.

Self-tracking will undoubtedly be used to oppress. It will wend its way into mainstream culture, eventually becoming something that employers expect of you as a matter of course. The temporal “productivity gaps” which we use to daydream, think about politics or other non-work related ideas, or simply consolidate memories, will be targeted and eliminated. Also, it is almost inconceivable that self-tracking data will avoid eventually going public.

Wolf gave his own response to some of the criticism (apparently a few minutes before Matt gave his).

I think many of the critical reactions make sense. What are we doing to ourselves? But I suspect that even the people who say something like “turn off the computer and go outside” are more deeply involved in the culture of self-tracking than they realize, and would benefit from going beyond initial revulsion. We _are_ in the process of changing. Our new selves will have new capacities as well as new vulnerabilities. Literacy itself was once a threat to our humanity: it interfered with memory, and substituted external representation for interior experience. It replaced living dialog with marks on a page. But we found a new sort of humanity in this world of letters.

The easy answer is that self tracking has to be done in moderation. Assuming it catches on, we’ll see public-service posters on buses and trains warning against overtracking and out-of-control “self love.” But every good thing is overdone and always will be. — and the solution has never been to ban it, deny it, or belittle it. It’s here, it’s coming, and we might as well use it.

See the article here, the 59 reader-recommended responses here, and all 138 online responses here. See the 7 letters here.

Back when secretaries were common, you could have had yours track your day in 15-minute increments. In his book The Effective Executive, Peter Drucker suggested this as a way to find out what you really did all day. The picture usually wasn’t so pretty.

Tracking your time then and now is personal, it’s messy, and it’s the essence of business intelligence: collecting data and reading it for guidance in business activities that matter. Is there anything that matters more to an organization than productivity of its people? For a small office or home-based business, this might be the best BI there is.

This gets no recognition in the BI industry that I can find, at least not in the conservative world of TDWI. At least not yet.

PI — for “private intelligence” — has different issues, starting with data collection. In BI, data comes from transactions, all recorded routinely. In PI, most of it has to come from a “secretary” or from our own, tedious notation.

I dabbled in it once. The insights were good, if painful, but mostly it was tedious. A few years ago, a confluence of personal events let me do something I’d always wanted to try: hole up for a few months in a Sicilian village I knew slightly. The food was good, I had relatives nearby, and the nearby church bells rang all day and all night, four times an hour. At the same time, I had a book to edit. To stay productive, I made a game out of the work, tracking my time to the minute in Filemaker.

I liked the local food and started to hate the book, an office manual that inadvertently revealed a con game. Even so, I threw myself at it every day. But no matter how hard I tried, no full day ever resulted in more than about two hours of actual, productive work. My “quick breaks” for walks and coffee with a friend actually took up more time.

I made a Filemaker database because I could find no off-the-shelf product that would do anything close. Each period, no matter how short, had a starting and ending times I entered with buttons, and a calculation field figured the duration. A drop-down menu offered my usual activites. I could make a report for any period.

I thought some product would do that better, but I could find nothing. Then the May 2 issue of the New York Times Magazine ran an article by Gary Wolf about this, “The Data Driven Life.” My Filemaker invention wasn’t too far from what others have used, and now new devices are coming along that could make all that seem so old hat. Some people are even sharing their data on the cloud.

But as in traditional BI, the technology just gets you in the door. The show has just begun.

Most people Wolf writes about do it for personal reasons. One wanted to know how his coffee consumption helped him focus, another tried to cure his sleep apnea, and still another noticed that flax seed oil, or just lots of butter, improved his cognitive performance.

As in good BI, the experiments often raised new questions. And sometimes the new questions are unexpected, as in Wolf’s own experience.

Often, pioneering trackers struggle with feelings of being both aided and tormented by the very systems they have built. I know what this is like. I used to track my work hours, and it was a miserable process. With my spreadsheet, I inadvertently transformed myself into the mean-spirited, small-minded boss I imagined I was escaping through self- employment. Taking advantage of the explosion of self-tracking services available on the Web, I started analyzing my workday at a finer level. Every time I moved to a new activity — picked up the phone, opened a Web browser, answered e-mail — I made a couple of clicks with my mouse, which recorded the change. After a few weeks I looked at the data and marveled. My day was a patchwork of distraction, interspersed with valuable, but too rare, periods of focus. In total, the amount of uninterrupted close attention I was able to muster in a given workday was less than three hours. After I got over the humiliation, I came to see how valuable this knowledge was. The efficiency lesson was that I could gain significant benefit by extending my day at my desk by only a few minutes, as long as these minutes were well spent. But a greater lesson was that by tracking hours at my desk I was making an unnecessary concession to a worthless stereotype. Does anybody really believe that long hours at a desk are a vocational ideal? I got nothing from my tracking system until I used it as a source of critical perspective, not on my performance but on my assumptions about what was important to track.

I wish Drucker were around to respond. Wolf’s insight sounds like important stuff for everyday knowledge workers, especially those who work alone. What’s more important to a knowledge worker than time?

These experiments are often haphazard and highly personal.

Generally, when we try to change, we simply thrash about: we improvise, guess, forget our results or change the conditions without even noticing the results. Errors are possible in self-tracking and self-experiment, of course. It is easy to mistake a transient effect for a permanent one, or miss some hidden factor that is influencing your data and confounding your conclusions. But once you start gathering data, recording the dates, toggling the conditions back and forth while keeping careful records of the outcome, you gain a tremendous advantage over the normal human practice of making no valid effort whatsoever.

Yes, just as analytics gives companies a tremendous advantage over those who make less effort.

Let the BI traditionalists pooh-pooh self-tracking. The very same people might have dismissed such things as visual analysis, agile development, and at one time even business intelligence itself. Sometimes it take a few pioneers and geeks, perhaps even a secretary, to prove a concept.

Here’s that eternal question again: how do you know when whatever you’re working on is good enough? Today, two perspectives.

The Oscar-winning sound designer Walter Murch, speaking Friday night at the Rafael Film Center in San Rafael California, told about talking shop with Michael Jackson’s engineers. They told Murch that they had no special insight, they just tried one mix after another as Jackson sat in the back, silent. They knew they had it right when he got up and danced.

The other way was the General Motors way. They took forever, and sometimes simply stopped trying when the bureaucracy’s deadline came. A recent New York Times article explains why GM cars never made anyone want to dance, or at least not me.

“We measured ourselves ten ways from Sunday … But as soon as everything is important, nothing is important.” [director of G.M.'s vehicle engineers]

Decisions were made, if at all, at a glacial pace, bogged down by endless committees, reports and reviews that astonished members of President Obama’s auto task force.

In the old G.M., any changes to a product program would be reviewed by as many as 70 executives, often taking two months for a decision to wind its way through regional forums, then to a global committee, and finally to the all-powerful automotive products board.

Nobody lives forever, but make sure you dance while you can.

The popular New York Times columnist and Nobel laureate Paul Krugman often disagrees with his fellow economists. Let’s all raise our eyebrows in shock.

Not! My shock is at the idea among some of his readers that there’s a right and wrong answer. Today, for example, Krugman again refers to his opinion that the stimulus passed early this year should have been bigger. To that, “Ray” comments:

I might suggest that Professor Krugman undercuts the supposed expertise of the economics profession every time he accuses a fellow Nobel laureate of not understanding elementary economics or being unable to read a graph. That sort of thing does not inspire confidence from the rest of us.

Guess what, Ray. Though there’s often a single version of the data, it’s much harder to agree on a single version of the truth.

Swap out a few terms in a recent New York Times story about farmers’ attempt to split California, and you might see the IT vs. business saga.

A quote halfway through caught my eye: “The agricultural industry is in this mode that says, ‘You will eat what’s put in front of you,’ and that’s a very condescending view of consumers and eaters. If customers are changing their preferences, the industry needs to change its ways.” Earlier in the story, an old farmer complains, “Those Hollywood types don’t have any idea what’s going on out here on the farms.”

Big Agriculture — affectionately shortened to Big Ag — seems analogous to the enigmatic Big BI. We can map food shoppers to business users, and back again.

The vast, quietly desperate majority settle for pre-packaged solutions, whether frozen dinners or standard reports. Most see little reason to change, or else they hate it all but see no way out. Locally grown produce? New ways of looking at the data? Yeah, right.

Some turn to alternatives like LucidEra, DataSelf and Birst, the on-demand chefs. They’re for people with food in the kitchen, of any origin, who don’t know what to do with it all or don’t have time.

Then we have the do-it-yourselfers. Those who insist they “need nothing fancy” prefer Excel, perhaps with the Veg-O-Matic add-in for slicing and dicing.

What to some food users is “fancy” is to others a cause to celebrate, and tools like Tableau Desktop and Lyza stay as close as the chef’s knife.

Tableau users are the Alice Waters of the data crowd: foraging local and remote sources, tasting, combining and trying out new concepts every day on friends in the kitchen and often at dinner parties. To them, data’s more than food, it’s the fire, too.

Lyza users also forage for eventual presentation, but at heart they may be more like old-time homesteaders. They thrash, mill and grind to extract the best they can get. They’ll wallow in muddy data if it means finding something better there. They’d rather do that than let someone else do it for them.

Meanwhile, back at Big Ag, the old farmer’s been out at shopping malls testing support for his initiative to split the state in two. He says, “I’m an old hound dog. If I’m barking up a tree, I want to know how many squirrels are up there.” I think he’s barking at data, not food.

Be sure you don’t miss Alex Vollmer’s excellent review of election day infographics. He wanted to see the margins of victory, percentage of precincts reporting, number of delegates at stake, and other goals. He looked at the New York Times online, CNN, National Public Radio and other media.