Getting Buried Alive by Data? Dig Out with Data Mining

The gap between being able to store data and process and analyze this data is growing at an exponential rate. If a company fails to put processes in place to close this gap, information overload will only get worse. Toward this end, data mining technologies and processes can effectively close the growing gap between the business decision maker and their data assets--helping companies to make informed actionable decisions.

Due to the economics of technology, corporations are able to collect and store information on nearly every interaction with their clients, with their suppliers and on their internal operations. In fact, according to a 2001 Forrester brief, Global 3500 enterprises spent, on average, $664,000 annually on database systems. But unfortunately, this ability to collect and store information lags behind the ability to effectively use it to support strategic business decisions.

If a company can leverage their data for decision support, they are more likely to have a competitive edge in their market sector. But the problem is that the very economics that allow corporations to collect and store details on interactions with clients, suppliers and their own processes are making it more difficult to leverage this data asset to strategically improve their business and their operations.

By and large, this problem emerges from a gap between computer processing power and new storage economics. On the one hand, "Moore's Law," which was predicted in 1964 by Intel co-founder Gordon Moore, states that computer processing power doubles every 18 months, for a fixed cost. As a result, we can easily purchase computers today with processing power that dwarfed the machines available even five years ago.

But on the other hand, computer storage manufacturers have outpaced their colleagues who build processors. The amount of computer storage that can be purchased for a fixed cost doubles roughly every nine months.

The effect of these computing "laws" and new storage economics are that the means of accumulating information and storing it far outstrip a company's capacity to process, sift through, analyze and use this data asset toward competitive business advantage. Moreover, as time passes, the gap between being able to store data (as measured by the amount of storage a corporation can obtain) and process, prepare and analyze this data (as measured by the amount of processing power a corporation can obtain), grows at an exponential rate.

If a corporation has not put processes in place to close this gap, information overload will continue. Their data assets quickly become data "tombs." In other words, data is simply stored and never sees the light of day.

This widening gap between computer storage and processing speed can be seen more clearly in the graph below.

Closing the Data Storage/Processing Gap

Experts who can design and build data storage and transformation systems that efficiently support high-level strategic analytic initiatives have played a leading role in closing the ever-increasing gap between a corporation's data and its strategic decision makers. These experts provide strategic business decision makers with solid, supportable data and trends that translate into improved customer and supplier relationship management and more efficient internal processes. Moreover, these improvements ultimately lead to improved competitive advantage.

Some forward thinking companies already are beginning to see their long-term investments pay dividends as they can now target their marketing and promotional efforts with unparalleled precision.

But most aren't. That's where the challenge comes in.

The Data Game: The Challenges Companies Face

The challenge is simple: how to extract the valuable knowledge from data, quickly and effectively.

Significant progress has been made over the last three decades by academic researchers in the fields of database technology, statistics and machine learning. This work formed the birth of data mining--a set of techniques and methodologies that can efficiently extract patterns and trends from large datasets.

By focusing these techniques and methodologies on a corporation's strategic business initiatives, the tried and true data mining technologies and processes are moving from the halls of academia into mainstream business practice. Data mining technologies and processes, when applied correctly, effectively close the growing gap between the business decision maker and their data assets helping corporations to make informed actionable decisions.

Historically, in-house technology staffs have been the only ones with clear access to collected data, while statistical analysts had the expertise to utilize complex statistical packages to do sophisticated manipulation and interpretation of the data. For corporations with the resources to support an IT staff and statistical analytic staff, data mining technology resided with these groups. But even in these situations, the ability to use trends and patterns extracted from data for improved marketing, pricing or targeting can be hampered due to political an other resource constraints.

What Does the Future Hold?

Despite the challenges of leveraging ever-growing stores of data, the good news is that over the next few years more and more companies are expected to put systems in place to benchmark the ways they measure and disseminate data. According to a recent META Group study, most companies will have adopted scorecards for tracking this information by 2005; and by 2007, leading organizations will undergo regular information audits.

An October 2003 Gartner study also found that Information democracy, corporate performance management and business activity monitoring are driving mass business intelligence adoption. But only about 35 percent of Global 3000 companies are aware that they need to maximize data. Another 45 percent of businesses operate on "reactive" data management mentality, used only in high-level strategic decisions and not spread throughout the entire business.

But this paradigm is clearly shifting.

Conclusion

Business intelligence initiatives continue to receive higher priority as companies realize the value of using their collected data to support strategic business decisions. In fact, the total business intelligence software market is forecast to see a compound annual growth rate of 8.5 percent by 2007, according to a Gartner study released this January.

Overall, this has also created a market opportunity for database and data mining experts. As more companies look to extract actionable knowledge from their data, these experts will be asked to build the technological infrastructures needed to effectively and efficiently support the data and analytic needs of corporate strategic decision makers.

---

Paul Bradley, Ph.D., is co-founder and principal, data mining technologies for Apollo Data Technologies, a Chicago-based firm that takes a consultative approach to data warehousing and data mining. In this role, Bradley directs specification, design and solution evaluation for data analysis problems. Prior to co-founding Apollo, he consulted on data mining algorithm integration with Microsoft Research and SQL Server, and led data analysis solution implementations for a number of Microsoft divisions. Earlier, Bradley was data mining development lead at digiMine, Inc., where he focused on integrating data mining technology into the company's service offering. Prior to that, he was a researcher in the Data Management, Exploration and Mining Group at Microsoft Research, where he helped develop new data mining algorithms and shipped data mining components in products such as SQL Server and Commerce Server.