One of our clients asked the question recently and got me thinking. We would like to believe that the world is always moving forward when it comes to data analytics. However, there are several reasons why that is not necessarily the case. I can think of at least four risks most companies are exposed to that pushes the data analytics maturity from time to time, in the wrong direction.
1. Technology Migration The first wave of data storage and processing for reporting and analytics was on plain vanilla relational databases. Then came the data appliances. Everybody started migrating to an appliance. Migrations took the center stage while the analytics projects were put on the back burner. Over many years, the data warehouses started maturating. Bang came Hadoop. It took few years, but now many are migrating to Hadoop. Every new company that jumps on to the new wave does not have the baggage, but all of the large mature organizations do.
Innovations in the storage and processing technologies are good. They make it faster, better and cheaper. The same innovations also make the legacy debt go higher and higher, data analytics maturity takes a pause or sometimes may even go backward.
2. Solution Obsolescence For many of the brick and mortar businesses the core business remains the same, while they find more efficient ways to do what they do, while finding new markets and new commercial opportunities. Take any large manufacturing company, they still have the factories and large supply chains. Some of the data analytics solutions aimed at making them more efficient become obsolete, just because the underlying technology is now considered legacy.
What if we could develop the solution in a technology agnostic manner? Then the investment in the solution will be protected. Even better if we could migrate the solution to a new technology seamlessly. Now that is future!
3. Skill Obsolescence Unfortunately, valuable skills related to data management, business knowledge, and quantitative skills are tied to the underlying technologies. People gets married to their favorite technologies. The transition of technology skills is not always easy or even possible. This is going to be the case until a new breed of data analytics solution development capabilities emerges that are technology agnostic.
One of the large companies I know started downsizing their EDS/BI team, while ramping up their Big Data/Data Science team. Just think about it, the business knowledge, data understanding and the quantitative skills just walked out the door. The incoming team has very little of all of those skills. Their data analytics maturity clearly took few steps back there.
4. Back to Hand Coding When the EDW first became popular, the tools to work on the EDW were not there. There were no code-free ETL tools, there were no drag-and-drop BI tools and no code-free analytical tools. Everyone wrote custom programs. Eventually, the tool sets matured and made the development more efficient and productive.
With Big Data now we are back to hand-coding in Java, Python, Scala, R and many other languages. Eventually, the toolsets will mature and hand coding will not be necessary, until yet another technology will come and make Hadoop a thing of the past. And the maturity is going to start over. Why do we have to be exposed to the same risks, over and over again? The new breed of tools will virtualize data analytics solutions future-proofing them. That will make the four risks I have mentioned in this post things of the past. Please see my previous post titled Data Analytics Factory Vision 2020 to have a glimpse of the future.