Webification of BI

MM: Then in history of business intelligence, the Web came along—and some things began to change. Could you quickly reprise us in terms of what changed how as a function of the Web, in the space of business intelligence?

MB: The Web changes everything. The Web changes some things directly and some things indirectly. One of the interesting forces in the database world and the data processing world is that the Web introduced a whole new realm of data to be handled.

The whole world of e-commerce introduced a need to understand e-commerce marketing, and to understand click-streams and how people were using the Internet and so forth. That created a number of new opportunities for people to try to process and understand the wealth of data, and to understand the customer behavior.

The companies that successfully handled Internet advertising have become the masters of this—Google and so forth. That’s the way that the Internet raised the stakes on this kind of marketing.

There’s also the absolutely direct benefit that the Web introduced—a new way to get information to people—in a way that is really much more appealing.

You’re able to get rid of many of the hassles and costs associated with software installation, if you can just give people a website to visit to get the information they’re looking for. People really like this model. It has all of the graphical capabilities that they’ve become accustomed to with their Office and installed desktop software.

That is an immediate thing that people latch on to: “Can’t I just have this on a web page, please?” Of course there is no reason that they can’t. There are a lot of companies like Oco making that happen now.

The Web also changes the way that the service, the calculations, and the data preparation can all be handled. Now, and throughout the history of data warehousing—going back to the mid-’90s, there was an awful lot of outsourced data warehousing. Lots of companies outsourced their data warehousing to big companies like Acxiom that specialized in data warehouse hosting, particularly for target marketing and related applications.

The Internet basically makes this idea a lot more attractive to companies—and in particular, attractive to companies with smaller budgets. It’s not just the big companies that can consider leveraging database and business intelligence technology, but in fact, everybody now can.

People are reluctant in some cases, because they fear, “Oh, gee, my precious data is going outside of my firewall.” But once people are satisfied that their data’s going to be handled securely, there are tremendous advantages.

One data-warehousing consultant I know said it pretty well, “All companies outsource the way their money is handled. That’s certainly precious to them. Why not data?”

MM: I think it’s because there’s a career track associated with it.

–

Data cubes got it started

MM: Again, we were in the middle of reprising the development of business intelligence. You’d talked about the early days of data warehouses and then how ERP started to move through a lot of corporations, normalizing a lot of that data, giving rise to the need for a master data management as a way of harmonizing data among systems.

Then I think you were about to launch into the emergence of business intelligence tools or technologies such as Business Objects or Cognos or Microstrategy or things like that.

MB: These tools, and the companies around these tools, emerged over time. There was a big flurry of tools companies that came into existence around this idea called OLAP or On-Line Analytical Processing. Its central idea was something called “Data Cubes” which allow you to analyze and manipulate data. They give you many different ways of looking at data and organizing it along different dimensions that you need to look at it. You could look at items by vendor, by price or by profitability or also by geographic region, organizational roles or hierarchy, etc. The “cube” notion comes by analogy to being able to turn a cube around in your hands to look at it from different perspectives.

These tools have been implemented in a variety of ways. In the early days, people had to summarize the data to a considerable degree in order to get these tools to perform very well. As computing power and storage has become less expensive, people have discovered that you really no longer need to summarize the data. In fact these tools become a lot more useful if you can actually drill all the way down to the lowest level of detail.

You can drill down all the way to the details, and observe issues associated with the data at finer granularity.  Then you are using the tool to figure out what’s causing the problem and how to solve it. This results in a much more flexible, robust, and efficient solution with much faster response times.

–

Critical success factor: Data model architect

MM: That almost reminds me of a conversation I had with a data warehouse architect. She was building a data warehouse for an executive information system for Bank of America. She would talk about sitting down with a fairly senior marketing executive and saying, “What are the business decisions that you make in the course of a day?” And then, “What information do you need in order to make a fully-informed decision?” And, “Where do you go for that information?”

Of course, there are green bar reports here and a conversation here and a fax here. In the course of doing that, she’d talk about identifying the most important—the number 1 or number 2 most important—business decisions that an executive would make. Then doing a map of logical but physical data sources, so as to be able to identify what the data items were that needed to be collated into information that then supported an action or an insight.

That kind of describes what you’re talking about in terms of this top-down optimization strategy or top-down problem-solving sort of thing.

MB: My expectation is that a large percentage of the projects that have been successful have had practitioners working on them in the model that you just described. Here at Oco, we’ve really taken that notion and turned it into an art form. We sit down with a business for one or sometimes two days and go through a systematic approach to define the key problems they need to solve. We call this approach a profiling session.

We design the solution and figure out the data resources that are going to be required and so forth. We have a quite robust methodology we go through. It’s a precise recipe.

–

Digital sail boats: Hole in the water in to which one pours money

–

MM: It sounds like a recipe for a very expensive digital sailboat.

MB: That’s what a lot of these projects are. That’s what has caused much of the difficulty and the high failure rate. There have been many successful data warehousing projects, but certainly a recipe to success is having some specific focus and purpose.

Many more benefits can accrue, but a lot of organizations simply run out of patience with the project before it has really gotten to the point where it’s delivering results.

At Oco, we do something quite different. I call it the top-down approach. We basically pick a business problem that is causing pain to the organization, and we identify a way of presenting the information to the business users in a way that we collectively believe will help them solve the problem.

We create this solution by bringing our best practices and knowledge of specific functions and industries to bear. Then we work top-down from this solution design to what specific data and related information sources need to be integrated to solve that problem.

So our integration work isn’t open ended. We know when we are done integrating.

–

Cross-integration of many system

MM: In fact, Mike—as we were doing the quick recap of the history of data warehouses… I think one of the things that had developed or emerged out of ERP and the data warehouses as they interact… A lot of the data that people need aren’t inside the organization. They’re outside the organization in suppliers and in trade partners.

MB: Well, that’s certainly the case. There’s information that’s down in the supply chain. But in fact, in the organizations we visit the systems that are facing the supply chains at customer sites are actually comprised of a variety of systems like warehouse management systems (WMS), transportation management systems (TMS), freight payment systems, customer relationship management systems (CRM), budgeting and financial planning systems, and so forth.

These systems provide functionality not found in ERP systems and therefore sit next to them—so companies always have many disparate, systems where important data resides and needs to be integrated for analysis and other purposes. There still needs to be some cross-system integration of this information. ERP systems in some sense have not quite lived up to their billing of consolidating all of this information. It remains important to be able to reach across different systems as well as across multiple ERP systems to be able to provide the visibility that companies need.

You mentioned the organizational barriers. Those are really quite significant as well. I’ll mention a couple of significant problems there. These are good examples for contrasting the way we approach these issues at Oco versus the way others in the industry have historically gone after these problems.

Historically, people will set out to put the data from their whole organization into the data warehouse. They’re trying to get data — all the data — in one place and also must cleanse that data—an enormous task. It’s an open-ended business intelligence activity that will enable the company to utilize the data warehouse…someday.

In other words, they’re building the data warehouse without already knowing exactly what they want to get out of it. They want to get whatever can come out of it.

–

–

Breaking points

–

MM: Could you give us a little bit of the history of business intelligence?

MB: Oco was formed to address the problems of existing BI tools, which were too difficult to develop and use. I can give you the historical perspective on that.

Back in the early 1990s, people started building data warehouses, because they didn’t have access to corporate information for the purposes of reporting and data analysis. They had lots of different operational systems, but they didn’t have systems that had data from all over the place gathered together.

These projects were originally pushing the relational database technology to the breaking point. Very large data warehouses were created, and every one of the vendors struggled to make these very large databases work.

But the software has matured now, allowing companies to put together quite, quite large data warehouses. There’s now an array of companies that offer BI tools. There’s also been some consolidation in the industry lately with SAP acquiring Business Objects and IBM acquiring Cognos and so forth.

Now there’s robust relational database software out there, and there are tools for accessing the information, but it has still been much too difficult. A recent report from Gartner estimates that still over 50% of these data warehousing or business intelligence projects fail.

–