Dan Murray’s taking on Big BI — and in just under two weeks at the Tableau Customer Conference in Seattle, he’s going to explain his four steps to rebellion — that is, “a high value, low cost BI reporting system.”
Dan devised the system when the company he worked for — which had revenue of about $70 million — couldn’t afford solutions from Big BI vendors. Bids ranged from $130,000 to $580,000.
Dan built his own with Tableau and Excel. The final cost, he writes, was $40,000 — $8000 for Tableau Desktop Pro and the rest for the database and ETL logic.
“I’m on a mission to spread this around the country,” he writes in email. “I consider to be every bit as big a revolution in data as the spreadsheet was to accountants in 1982. It’s thrilling.”
In response to Chris Tyler….
While big BI has done the world a service through the good works of Inmon, Kimball and others, the typical Big BI installation is simply too costly for most business establishments to take on.
My specific “ah-ha” moment occurred when I was working for a mid-sized global manufacturing company a couple of years ago. I got the quotes from several vendors and the price to implement for the same specification ranged from $130,000 to $580,000. Simply too costly.
The primary cost drivers included:
a. Vendor software (around 30% to 40%) of the total quoted cost.
b. Consulting time for data cleaning (ranged from 25% to 40%) of the total time.
In addition, each quote required that I agree to provide a specific number of hours of my time to support the effort in two areas:
a. Assist in the data cleaning work (the consultants couldn’t do this without significant input from internal staff).
b. Developing the ETL logic for data validation prior to loading into our main database repository.
It occurred to me that If I had to be directly involved for the data cleaning I should probably take on the entire project myself and save the company cash for other things.
Now, I’ll be the first person to say if you run a bank, insurance company or a large consumer product company with millions of transactions per day….proceeding with a “thrilling rebellion” and spreadsheets is NOT the way to go. But you already know that. Your probably one of the 2% to 3% of the business establishments in the world that fit that description. You also probably already have a data warehouse and millions invested in the setup and upkeep of that valuable asset.
On the other hand, if you happen to be one of the other 97% to 98% of the business, municipal or non-profit entities operating, the approach that I suggest is completely safe.
I suggest that Tableau Software with an evolutionary migration using spreadsheets first (replicating a star schema model in the spreadsheet using what I call a “dimensionally-conformed” spreadsheet) provides a fast and safe path to paving the way for your own full-scale data warehouse without 90% of the cost of the traditional data warehouse implementation.
I know it works because I successfully implemented my own data warehouse using this approach. My initial cost was $0.00 plus a personal time investment of 80 hours. With that time and a free trial copy of Tableau Desktop Professional I created all the reports that the President of my company wanted.
This lead to an eventual migration to a MS SQL Server 2005 database back end. The total eventual cost ended up being around $40,000, but I had reports in the first 30 days that met all of my users’ needs via Tableau. Within 9 months I had a full data warehouse with ETL logic in North America and Europe. I only used outside consultants to do the data warehouse architecture and scripting of The ETL logic. To be fair I should disclose that I already had an underutilized copy of MS SQL server running for other applications, so I didn’t need to invest the additional $8,000 – $15,000 that would have been required to obtain a legal copy of that software.
My final cost tally broke down this way:
a. Tableau software: $10,000
b. Data Architect: $30,000 (to build-out the schema and ETL that I provided)
I didn’t pursue the data warehouse option until we (my internal team o 3 – myself included) had worked on data quality while producing weekly/monthly data visualizations using Tableau with dimensionally-conformed Excel spreadsheets. We did all dynamic data visualization distribution via Tableau’s Free Reader product for 5 months all the while honing our ETL logic manually.
When I asked for the additional funding for the data warehouse the response was immediate. DO IT! There was no capital appropriation needed. The value of what we did was already clearly demonstrated. I just asked the present if he wanted to be able to play with an unlimited amount of history versus a monthly series of reports.
This approach is attainable to the millions of smaller businesses around the world who simply can’t justify the large investment that current Big BI has to offer. The other advantage of the approach I suggest is a much greater understanding and ownership of the entire project by internal staff and management resources.
By all means though….if you’re a large company and have millions of transactions per day to sort out…and the cash to do it, hire a Big BI consulting firm to help you implement a data warehouse. If you don’t already have one your way behind the curve.
So while I think Chris is right for very large or complex entities I respectfully disagree with his conclusion. We didn’t have data silos.
I do agree that clients of Big BI companies would benefit from the use of Tableau Software and would love to participate with Chris or any of his clients to help them get value from Tableau. As a data cleaning and report prototyping tool it’s excellent.
I quit my job of 17 years to help other people realize the benefits of High Value Low Cost BI. If you have any interest please contact Dan Murray at http://www.interworksinc.com.
Full Disclosure: I am an employee of a “Big BI” vendor, one of the biggest actually.
I have been following Tableau for a while and am quite impressed with it’s Data Visualization capabilities. For an end user wanting to analyze and understand their data, it is an easy to learn and easy to use tool. The interactivity with the data is great. That’s where it ends. Tableau is a terrific Data Visualization tool.
I don’t think you should be looking at Tableau as a replacement or competitor of “Big BI”, but more as complementary technology. “Big BI” is more about metadata and being able to leverage the same data in multiple visualization tools. Tableau should be one of those visualization tools that leverage the “Big BI” metadata and data.
“Big BI” vendor software applications allow users to create what-if scenarios, ad-hoc reports, professional and analytical reports, analyze through slice-and-dice, create alerts and notifications, develop scorecards (which go well beyond stop light reports) and deliver analytical applications. Because of the metadata structure, all of the numbers are the same, regardless of the visualization tool.
Without the rich, underlying metadata structure, tools like Tableau end up replicating business logic and have large volumes of silo’ed data in excel spreadsheets being independently managed. Who’s numbers are right? I can appreciate the visualization capabilities of Tableau, I would recommend a cooperative or co-opetitive approach not a competitive one.
I love it. A) the realization that data analysis just should not cost that much and B) the ratio of front end (visualisation) to back end (data and ETL) – 80% getting the data in a form it can be looked at, 20% looking at the data. Not to say that having fantastic visualisation like Tableau is not key its just that so much data needs so much work that ETL always takes a serious amount of effort, yet the focus of Big BI tends to be on the dashboards and front end stuff (which they don’t do anywhere near as Tableau last time I checked). Hope you do a write up of the talk, would love to have more info.